Re:Appropriate Uses for SQLite
by hawski on Monday September 26, 2016 @04:51AM
(#12580291)
I would like to see more use of netstrings - https://en.wikipedia.org/wiki/Netstring
[ Parent
| on HN ]
This is a bad idea. What you want from text logs is the possibility to start anywhere in the file, look for a nearby newline and know it is the start of a record.
This is a very important property. It's also what makes UTF-8 resilient.
Why can't I have both? I am not against text logs, but it is good to have options. When binary format is an option I say that netstrings are probably better option.
For literal binary data you need to store the length information separately.
Netstrings are one option for ephemeral streams. But they are a bad tradeoff for persistent information: you have to read from the beginning of the stream until you find the relevant information. And a single corrupted byte destroys all the information after it.
Better options for persistent data are (pointer+length)s or memory-pools + ranges.