Re:Parsing can become accidentally quadratic because of sscanf
by minitech on Monday March 01, 2021 @04:17PM
(#26308531)
Text is the problem, here and everywhere else. Text encourages looking for delimiters, escaping, and flexibility (optional whitespace, case insensitivity, leading zeros, redundant signs, newline style compatibility). Compared to binary with length prefixes, that’s really slow, prone to injection vulnerabilities and subtle incompatibility, requires copying, introduces more side channels (the length of your message now depends on the values of numbers inside it!), ….
Re:Parsing can become accidentally quadratic because of sscanf
by CivBase on Tuesday March 02, 2021 @09:55AM
(#26315922)
Text =/= Variable Length
I can store fixed-length data using text. A strict schema can specify things like fixed lengths, value padding, orders for maps, exact whitespace rules, etc which can entirely remove the need for delimiters. A schema could even require length prefixes, just like you might use for a non-text format.
These concepts do not necessitate abandoning text. If you need the data to be readable/writable by a human with minimal tooling, you can still leverage text without sacrificing much parsing efficiency.
Re:Parsing can become accidentally quadratic because of sscanf
by minitech on Tuesday March 02, 2021 @03:49PM
(#26320457)
In theory, sure. In practice, nobody does, and length prefixes stop a format like that from being conveniently writable with a plain text editor.
Re:Parsing can become accidentally quadratic because of sscanf
by CivBase on Tuesday March 02, 2021 @07:19PM
(#26323125)
I work on avionics and we definitely use fixed-length text all the time. Text is often much more convenient than blobs, even when adhering to strict schemas.
Syntaxes like XML, JSON, YAML, etc have become popular for their convenience and versatility, but they do not represent the whole of what is possible with text.
Re:Parsing can become accidentally quadratic because of sscanf
by renox on Thursday March 04, 2021 @08:50AM
(#26342346)
You've not used enough Ada: its variable length string is so clunky that fixed sized text is used instead..
Re:Parsing can become accidentally quadratic because of sscanf
by deathanatos on Monday March 01, 2021 @11:20PM
(#26311874)
And once you get down to it, text still requires tools to parse. I routinely run stuff through `jq`, because despite JSON being a text format, actually consuming it as a human requires a tool. (Often to filter it, but sometimes just to colorize the output to make it legible.)
I don't think text is necessarily the problem here. Other protocols might allow you to use less memory, but they aren't running into a memory limitation in this instance.
You can divide parsers into two groups: parsers designed to read data which follows a specific schema and parsers designed to read arbitrary data.
Schema-aware parsers have additional context which allows them to process the data much more efficiently. The stricter the schema, the more efficient the parser can be. Meanwhile, parsers for arbitrary data have no such context and must rely completely on syntax to parse the data. A poorly optimized schema-aware parser may still outperform a well optimized parser for arbitrary data.
You can still encode data as text and even use popular syntaxes like JSON or YAML with schema-aware parsers. I've personally helped design a schema-aware parser for XML data. But you should keep in mind that most third-party parsers for those syntaxes will be designed to parse arbitrary data. Parsers for arbitrary data can still be very fast - fast enough for many purposes. However, schema-aware parsers have a significant advantage when it comes to computational performance.