64 bytes would be a 512-bit integer. That seems like excessive future proofing for the length of any message that would be transmitted before the Sun runs out of fuel.

Not my thought originally, I heard it from somewhere else but can't find it. Possibly from Foone Turing.

[1] https://www.wolframalpha.com/input?i=log2%28+volume+of+unive...

Plus rather simple things like pi could create rather a long message.

All digital representations rely on discrete states in hardware, and there’s a finite number of those in the observable universe, so there should a finite maximum number for computers.

Just as a quick example of why it's a bit absurd - I name that number you just defined $zeta$. Now I make $zeta'$ = zeta^zeta. Or whatever manipulation you like. Adding constraints is addressed in the link.

The GP question was not about encoding, and thus is not subject to compression. The largest number we can measure of anything is a pretty well defined concept.

We don’t know if there is only a finite number of discrete states in the observable universe. The Planck length is not a discretization.

I'll have to think about this. I want a very physical meaning of the term using values rather than references. One where the largest number that can be expressed using up/down fingers with two hands is 1024, not a sign language reference to a googolplex.

"Measuring the intelligence of an idealized mechanical knowing agent" https://philpapers.org/archive/ALEMTI-2.pdf

"Intuitive Ordinal Notations" https://github.com/semitrivial/IONs

https://en.m.wikipedia.org/wiki/Hyperoperation

I wonder if it would be helpful to restrict the question a bit, maybe something like: what is the largest number for which the number, and also all smaller magnitude integers, can be expressed.

You can, of course, simply read field by field, but few libraries expose the ability to do that simply. And the naive operation then becomes quite problematic indeed.

The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards for doing this:

- SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/go...

- Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli

    81 A2 70 62 C6 <32 bit be length> <protobuf data>

0x8n A map containing "n" key/value pairs -> 0x81 introduces a single key/value pair

b101XXXXX : A string where XXXXX is the string length -> 0xA2 introudces a 2 byte string

0xC6 followed by a 32-bit value N introduces an array of bytes of size N

0x70 0x62 "pb"

So it roughly corresponds to

  {"pb": <protobuf data> }

/s

The only issue is if you were to ship a “protobuf” library that emits/consumes your (very much not protobuf) framing format.

Also, a 64 bit frame length would only be 8 bytes, not 64 :-)

In general, I find adding versions to things allows for much more graceful future redesigns and that is, IMO, invaluable if you're concerned about longevity and are not confident in your ability to perfectly design something for the indefinite future.

The way protobufs handles versioning (old software ignores unknown fields) is far superior and realistically everyone uses 64 bit fixed length sizes everywhere

The idea is so simple that any change would be a misfeature.

The magic means it’s possible to identify the file type. Maybe you’ll add a tool later that operates on multiple types of files.

The version means you can evolve the contents in non-backwards-compatible ways, while maintaining the ability to read/parse the old version.

It’s a pain in the ass to add a magic or version number later; there’s a reason why nearly every file format on the planet has both.

XML

JSON

YML

TOML

  <?xml version="1.1" encoding="UTF-8" ?>

> they're not file formats

Brb, gonna delete all my files without file formats

If you’re going to use text-based serialization formats as your justification for the decision, however, I’d suggest you look into all the fun bugs, security issues, and weird edges cases that arise from parsers having to make a best guess at character encoding and file format when all you have to work with is the file extension, maybe a byte order mark, and heuristics over the file contents.

Because that's all you would need.

https://protobuf.dev/reference/java/api-docs/com/google/prot...

https://clickhouse.com/docs/en/sql-reference/formats#protobu...

Using a delimiter to mark end of packet. For text protocols we usually use "\n" to be the delimiter. This usually requires some escaping in the packet to make it unambiguous. Two standard protocols for this are SLIP and HDLC.

Length encoding - like what you did.

The downside of the delimiter approach is that it changes the length of the packet - when you are escaping one byte becomes two and that's sometimes a pain if you're doing it in place in memory (less of a problem if you're streaming byte by byte). The big advantage is that it allows for resynchronisation - if you lose a single byte from your stream, or your length byte ever gets corrupted then you're permanently out of sync - the receiver will never again know where the start or end of a packet is. With the delimiter approach, you just lose one packet. So if you're ever doing this for a UART or network stream or something, always do the delimiter approach!

(The TCP checksum is very weak at only 16 bits, but there's usually another layer like TLS that gives you integrity for free so nobody cares)

I'm confused. I mean for UART, sure. But network streams is usually sent over a protocol that recovers lost data. Am I reckless for sending length-prefixed data chunks over TCP?

I actually use SLIP for packetisation over TCP anyway, because then syncing for any other logging or diagnostic thing that joins halfway through is easy. Basically I find it to just be a more robust system.

Whoa, that's an acronym I haven't heard in a long time.

SLIP/TCP seems redundant. Your serial encoding IP then sending it over TCP/IP? Is this a tunnel?

You're right that it was originally for segmenting a serial link to send separate IP packets (so you could theoretically have SLIP over TCP over IP over SLIP), but it works just as well in other contexts.

Shove protobuf into Something Else that does packet delimitation for you. I'm fond of SQLite for offline cases as a richer alternative to sstable.

I shudder to think how well shit must be going for this to merit a Rachel post.