News for hackers. Stuff that matters.
Re:Varlink – IPC to replace D-Bus gradually in systemd by deivid on Sunday September 29, 2024 @02:11PM (#41689194)
There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging.

[0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

Re:Varlink – IPC to replace D-Bus gradually in systemd by anotherhue on Sunday September 29, 2024 @04:15PM (#41690076)
With this logic (oft repeated) we should be sending TCP as JSON. { sourcePort: 443, destinationPort: 12345,...}

Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.

Re:Varlink – IPC to replace D-Bus gradually in systemd by brookst on Sunday September 29, 2024 @05:24PM (#41690759)
I love your TCP over JSON idea. Maybe we can implement layer 2 that way as well, with CSMA/CD: {startTime:2024-09-29T13:56:21.06551, duration:00:00:00.0024, packet: …}
Re:Varlink – IPC to replace D-Bus gradually in systemd by seba_dos1 on Monday September 30, 2024 @03:06AM (#41694336)
You forgot quotes. JSON has no date or time types :)
Re:Varlink – IPC to replace D-Bus gradually in systemd by gmokki on Sunday September 29, 2024 @02:26PM (#41689314)
I do not understand where the 64bit integers not working with JSON comes from.

JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing.

Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields.

Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.

Re:Varlink – IPC to replace D-Bus gradually in systemd by capitainenemo on Sunday September 29, 2024 @03:21PM (#41689681)
The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1.

BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

Re:Varlink – IPC to replace D-Bus gradually in systemd by IshKebab on Sunday September 29, 2024 @05:56PM (#41691071)
That's not true. The number type in JSON is an arbitrary sized number. You're thinking of JavaScript which is not the same thing.
Re:Varlink – IPC to replace D-Bus gradually in systemd by capitainenemo on Sunday September 29, 2024 @06:59PM (#41691608)
https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No.

(which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6)

If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed.

(I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)

Re:Varlink – IPC to replace D-Bus gradually in systemd by skissane on Sunday September 29, 2024 @07:55PM (#41692007)
You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements.

JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON

Re:Varlink – IPC to replace D-Bus gradually in systemd by capitainenemo on Sunday September 29, 2024 @08:00PM (#41692028)
Yes, they are different standards, but the purpose of I-JSON was specifically to solve interoperability problems. Those interoperability problems are due to the looseness of the original spec.

The others do not forbid larger numbers but note that doing that will make for bad times, and if you look at JSON parsing matrices (say, the "Parsing JSON is a Minefield" one, you can see that the majority of implementations do not do allow larger). So. float64 is a defacto standard and I-JSON is simply clarifying this. Given the main purpose of JSON is an interoperable data exchange format it would be a very bad idea to do otherwise.

https://en.wikipedia.org/wiki/JSON?useskin=vector#Interopera... (essentially same point made here)

Re:Varlink – IPC to replace D-Bus gradually in systemd by dathinab on Monday September 30, 2024 @03:54AM (#41694619)
yes but no

there is more then one JSON standard and while the JS in JSON stands for JavaScript it doesn't directly means it is using JS types as confusing as it might seem.

the initial definition of JSON by itself has no limit on integer size at all, JSON with number which can only be represented with BigInt _totally is a thing_ just not so much in the Js ecosystem. Furthermore systems accidental expecting full i64 or u64 number ranges are as much a thing, and this leading to subtle bugs the moment a different (de-)serializer is used is another issues.

Re:Varlink – IPC to replace D-Bus gradually in systemd by GrayShade on Sunday September 29, 2024 @02:46PM (#41689456)
Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
Re:Varlink – IPC to replace D-Bus gradually in systemd by Spivak on Sunday September 29, 2024 @03:31PM (#41689748)
jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
Re:Varlink – IPC to replace D-Bus gradually in systemd by capitainenemo on Sunday September 29, 2024 @07:23PM (#41691792)
Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
Re:Varlink – IPC to replace D-Bus gradually in systemd by GrayShade on Saturday October 12, 2024 @03:17AM (#41817094)
No, it doesn't. It's not a SHOULD.

And why should Qt be concerned with what jq can or can't do? If you're writing and reading back from Qt, the limitations of other implementations don't apply.

Re:Varlink – IPC to replace D-Bus gradually in systemd by theamk on Sunday September 29, 2024 @03:33PM (#41689755)
If you fully work within a trusted environment, why bother with JSON? Use your favorite binary serialization with codegen.

The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.

Re:Varlink – IPC to replace D-Bus gradually in systemd by djbusby on Sunday September 29, 2024 @08:04PM (#41692051)
Cool, readable messages in strace but still some odd binary log format?
Re:Varlink – IPC to replace D-Bus gradually in systemd by criticalfault on Monday September 30, 2024 @05:33AM (#41695227)
Re:Varlink – IPC to replace D-Bus gradually in systemd by quotemstr on Sunday September 29, 2024 @11:25PM (#41693247)
Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools.

Christ almighty, this is a terrible proposal.

Re:Varlink – IPC to replace D-Bus gradually in systemd by zbentley on Monday September 30, 2024 @09:14AM (#41696750)
The observability tools in question (strace) follow the UNIX tradition of displaying and processing data as text by default. I don’t think that means that they suck.

I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable.

The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge.

I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.