Appropriate Uses for SQLite

Re:Appropriate Uses for SQLite by sebcat on Monday September 26, 2016 @04:34AM (#12580198)

For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of failure when accepting input from external sources. There should be no difference between:

    {"foo": 1}{"bar": 2}

and

    {"foo": 1}
    {"bar": 2}

or any variations of common whitespace between the objects. Just parse objects incrementally, one at a time. Most JSON libraries supports this.

As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case.

Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.

[ Parent | on HN ]

Re:Appropriate Uses for SQLite by tracker1 on Monday September 26, 2016 @05:48AM (#12580503)

It's simply easier to determine when an object ends with a delimiter...

    {"dtm":"2016-...",...,meta:{...}}

That requires more complicated input processing/streaming than read until '\n' then parse the record...

[ Parent | on HN ]

Re:Appropriate Uses for SQLite by eatonphil on Monday September 26, 2016 @07:43AM (#12580997)

Not meaningfully more complicated. You know when you've reached the end of your object of you are actually parsing it, so you also know when the next one can begin.

[ Parent | on HN ]

Re:Appropriate Uses for SQLite by JdeBP on Tuesday September 27, 2016 @12:29PM (#12590840)

Actually, no, one is not necessarily parsing it. Xyr point was that using line delimiters allows one to retain one-log-record-per line semantics and use the log stream with text processing tools that understand those semantics but that do not understand JSON.

[ Parent | on HN ]

Re:Appropriate Uses for SQLite by tracker1 on Monday September 26, 2016 @01:08PM (#12583584)

When you're working with a platform that has native JSON parsing, it generally doesn't support streams as described... so you either count braces and quotes, create a stream processor or use a delimiter... a delimiter is an easy enough solution, also allows for easier viewing in a text editor. It's about being pragmatic.

[ Parent | on HN ]