News for hackers. Stuff that matters.
Re:Add extra stuff to a “standard” encoding? Sure, why not by tetrep on Wednesday September 20, 2023 @07:58AM (#37583102)
If you want to future proof it you need to version it. It sounds like you're trying to pack many things into a single file, so having the first few bits of the file represent a version allows you to use fixed length integers without fear of them being too small (in the future). You can reserve the "last" version for varint if you truly need it.

In general, I find adding versions to things allows for much more graceful future redesigns and that is, IMO, invaluable if you're concerned about longevity and are not confident in your ability to perfectly design something for the indefinite future.

Re:Add extra stuff to a “standard” encoding? Sure, why not by NavinF on Wednesday September 20, 2023 @08:28AM (#37583321)
I don't see the point. As soon as you bump the version number, old versions of the software will refuse to read newer files. So you might as well use a new format and file extension for the newer files. Of course the new software will read both formats.

The way protobufs handles versioning (old software ignores unknown fields) is far superior and realistically everyone uses 64 bit fixed length sizes everywhere

Re:Add extra stuff to a “standard” encoding? Sure, why not by paulddraper on Wednesday September 20, 2023 @08:10AM (#37583204)
YAGNI

The idea is so simple that any change would be a misfeature.

Re:Add extra stuff to a “standard” encoding? Sure, why not by stravant on Wednesday September 20, 2023 @08:19AM (#37583267)
I'm a big YAGNI proponent but format version numbers are not something I would dare to YAGNI.
Re:Add extra stuff to a “standard” encoding? Sure, why not by paulddraper on Wednesday September 20, 2023 @09:12AM (#37583698)
Even if it's varint separated messages?
Re:Add extra stuff to a “standard” encoding? Sure, why not by catiopatio on Wednesday September 20, 2023 @09:20AM (#37583774)
You stick a magic number and a version number at the top of the file and call it done. It’s trivial and buys you a great deal.

The magic means it’s possible to identify the file type. Maybe you’ll add a tool later that operates on multiple types of files.

The version means you can evolve the contents in non-backwards-compatible ways, while maintaining the ability to read/parse the old version.

It’s a pain in the ass to add a magic or version number later; there’s a reason why nearly every file format on the planet has both.

Re:Add extra stuff to a “standard” encoding? Sure, why not by paulddraper on Wednesday September 20, 2023 @11:15AM (#37585267)
> nearly every file format on the planet

XML

JSON

YML

TOML

Re:Add extra stuff to a “standard” encoding? Sure, why not by catiopatio on Wednesday September 20, 2023 @12:31PM (#37586276)

  <?xml version="1.1" encoding="UTF-8" ?>
The others are serialization formats, like protobuf — they’re not file formats.
Re:Add extra stuff to a “standard” encoding? Sure, why not by paulddraper on Wednesday September 20, 2023 @02:16PM (#37587759)
XML prolog is optional.

> they're not file formats

Brb, gonna delete all my files without file formats

Re:Add extra stuff to a “standard” encoding? Sure, why not by catiopatio on Wednesday September 20, 2023 @07:56PM (#37591622)
If you prefer to ship fragile binary file formats for no reason other than finding it too onerous to define something as trivial as an eight-byte file header, that’s silly, but nobody is going to stop you.

If you’re going to use text-based serialization formats as your justification for the decision, however, I’d suggest you look into all the fun bugs, security issues, and weird edges cases that arise from parsers having to make a best guess at character encoding and file format when all you have to work with is the file extension, maybe a byte order mark, and heuristics over the file contents.