News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by ryandrake on Monday April 29, 2024 @07:03PM (#40205194)
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

What ever happened to the Robustness Principle[1]? I think the entire comment section of this article has forgotten it. IMO the best software accepts many formats and "deals with it," or at least attempts to, rather than just exiting with "Hahah, Error 19923 Wrong Input Format. Try again, loser."

1: https://en.wikipedia.org/wiki/Robustness_principle

Re:You can't just assume UTF-8 by jerf on Monday April 29, 2024 @07:12PM (#40205273)
We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded. This is in some sense relatively recent, so word is still getting around, as the programming world does not move as quickly as it fancies itself to.

If you'd like to see why, read the HTML 5 parsing portion of the spec. Slowly and carefully. Try to understand what is going on and why. A skim will not reveal the issue. You will come to a much greater understanding of the problem. Some study of what had happened when we tried to upgrade TCP (not the 4->6 transition, that's its own thing) and why the only two protocols that can practically exist on the Internet anymore are TCP and UDP may also be of interest.

Re:You can't just assume UTF-8 by fanf2 on Monday April 29, 2024 @07:40PM (#40205529)
RFC 9413 “maintaining robust protocols” https://datatracker.ietf.org/doc/rfc9413/ was originally titled “The Harmful Consequences of the Robustness Principle (draft-thomson-postel-was-wrong)” https://datatracker.ietf.org/doc/draft-thomson-postel-was-wr... and it has several examples
Re:You can't just assume UTF-8 by tomsmeding on Monday April 29, 2024 @07:36PM (#40205492)
Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec? Or is it genuinely unexplainable without experiencing something first-hand?
Re:You can't just assume UTF-8 by fl7305 on Monday April 29, 2024 @07:43PM (#40205566)
>>> the best software accepts many formats and "deals with it,"

>> We collectively discovered that we were underestimating the long term costs, by a lot, so its lustre has faded.

>> If you'd like to see why, read the HTML 5 parsing portion of the spec.

> Can you give a hint as to what the issue is that one should find reading a portion of the HTML 5 spec

I think the point was that the HTML 5 spec tries to parse all kinds of weird input instead of drawing a line in the sand and forcing the input to follow a simple format?

Re:You can't just assume UTF-8 by Spivak on Monday April 29, 2024 @08:20PM (#40205889)
It's ungodly complicated to be sure but websites from 30 years ago also still render unmodified so it's hard to say it's bad all of the time.
Re:You can't just assume UTF-8 by jerf on Monday April 29, 2024 @09:01PM (#40206162)
It isn't bad. In fact it's quite good. But it is very much a case of closing the barn door after the animals got out. You see in the standard that the effort was put to corral them back in, and I'm very glad they did, but it certainly was not free.
Re:You can't just assume UTF-8 by riffraff on Tuesday April 30, 2024 @12:30AM (#40207258)
Doesn't HTML5 prove that such an approach was more effective tho?

HTML was a massive success, while at the same time when we tried having a strict spec with XHTML, we failed miserably.

I mean, the purist in me wants strictness but it seems lousyness wins.

Re:You can't just assume UTF-8 by kelnos on Tuesday April 30, 2024 @01:26AM (#40207520)
I think the particular domain matters.

The explosion of the web happened in no small part because of how easy it was to write some HTML and get a basic, working webpage out of it. If you nested some tags the wrong way and the browser just put up an error page, rather than doing a (usually) pretty good job figuring out what you actually meant, people would get frustrated faster and not bother with it at all.

But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.

The difficulty in drawing a line in the sand and sticking to the spec, though, is that of user blame. Let's say you implement a spec perfectly -- even if you are the originator of the spec -- and then someone comes along and builds something of their own that writes out files that don't conform to the spec. Your software throws up an error and says "invalid file", but the other piece of software can read it back in just fine. Users don't know or care about specifications; they just know that your software "doesn't work" for the files they have, and the other software does. If you try to tell them that the file is bad, and the other software has a bug, they really won't care.

Re:You can't just assume UTF-8 by AnthonyMouse on Tuesday April 30, 2024 @06:54AM (#40209553)
A possible solution to this is for large organizations to be intransigent about standards compliance. If your personal mail server rejects mail that isn't well-formed, you're just being a masochist because nobody is going to change for you. If Google does it, everybody else is going to fix their stuff because otherwise they can't send to gmail.
Re:You can't just assume UTF-8 by astine on Tuesday April 30, 2024 @09:52AM (#40210954)
But imagine if our C/C++/Java/Rust/Go/etc. compilers were like "syntax error, but ehhhhh you probably meant to put a closing brace there, so let's just pretend you did". That would be a nightmare of bugs and security issues.

That's how Perl ended up the way it is.

Re:You can't just assume UTF-8 by jerf on Tuesday April 30, 2024 @10:14AM (#40211150)
It proves that engineering is hard and that everything has costs and benefits, and you can't make them go away by just ignoring them. It turns out that "being robust" had a lot of costs we didn't see.

It also shows the harmfulness of binary black and white thinking in engineering. There are choices other than "just let everyone do whatever and hope all the different things picking up the pieces do it in more or less the same way" and "rigidly specify a spec and blow up the universe at the slightest deviation". Both of those easy-to-specify choices have excessive costs. There is no escape from hard design tasks. XHTML may always have been doomed to fail, but that is not to say that HTML had to be allowed to be as loosey-goosey as it is, either.

Had a gradient of failure been introduced rather than a rigid rock wall, things very likely wouldn't have gotten as badly out of hand as they did. If, for instance, a screwed up table was specified to deliberately render in a very aesthetically unappealing manner, but not crash the entire page the way XHTML did, people would have not come to depend so much on HTML being sloppy. The resulting broken page would still be somewhat usable, but there would have been motivation to fix it, rather than the world we actually live in where it all just seemed to work.

Re:You can't just assume UTF-8 by travisb on Monday April 29, 2024 @07:20PM (#40205350)
Among other things, security became a concern.

Being lenient is all well and good when the consequences are mild. When the consequences of misinterpreting or interpreting differently to a second implementation becomes costly, such as a security exploit, then the Robustness Principle becomes less obviously a win.

It's important to understand that every implementation will try to fix-up formatting problems in their own way unique to their particular implementation. From that you get various desync or reinterpretation attacks (eg. HTTP request smuggling).

Re:You can't just assume UTF-8 by thfuran on Monday April 29, 2024 @07:27PM (#40205420)
It was a horrible idea. The real robustness principle is "Follow the spec".
Re:You can't just assume UTF-8 by nomel on Monday April 29, 2024 @07:31PM (#40205455)
I've tried telling users "sorry, your file isn't to spec", and they say "but it works with <competitor>", and that ideology flies right out the window, along with their money.
Re:You can't just assume UTF-8 by ryandrake on Monday April 29, 2024 @08:13PM (#40205825)
Exactly. "Accepting and trying" is how a lot of popular software won their market. Look at HN's favorite media player, VLC. In the past, media player software were horrible, refusing to play all but the most tightly constrained set of allowed containers/codecs. I remember spending the early 2000s trying to get Windows Media Player to play MPEGs by downloading codec packs and trying to cast secret spells into the Windows Registry. Yuck! Then along comes VLC which accepts whatever you throw at it, and that software is basically everywhere now. You can throw line noise at VLC and it will try its best to play something!
Re:You can't just assume UTF-8 by pixl97 on Monday April 29, 2024 @08:11PM (#40205807)
That's when you ask to see their file, then hack <competitors> system.
Re:You can't just assume UTF-8 by thfuran on Monday April 29, 2024 @08:56PM (#40206133)
People will pay you to do all kinds of terrible not-robust things.
Re:You can't just assume UTF-8 by Joker_vD on Tuesday April 30, 2024 @03:56AM (#40208336)
Yes, and that's why Postel's law is more of an empirical observation (a law of nature, if you will) on which software survives and which doesn't. You may dislike it but that won't make it go away.
Re:You can't just assume UTF-8 by nomel on Wednesday May 01, 2024 @02:27PM (#40227223)
I see it as the opposite, actually. They will pay you for a robust product: one that works for them. They have no care for the technical minutiae of your implementation, because they're just trying to do actually interesting things, with the help of your product. This is the customer perspective.
Re:You can't just assume UTF-8 by silotis on Monday April 29, 2024 @09:01PM (#40206164)
The trick is to enforce conformance right from the start in the first implementation of a format. Shipping a product that doesn't interop with the existing tools is a non-starter so the devs will have to fix their shit first.

As you say, unfortunately the genie cannot be put back in the bottle for formats that already have defective implementations in the wild.

Re:You can't just assume UTF-8 by riffraff on Tuesday April 30, 2024 @12:51AM (#40207360)
The problem is that unless you restrict everyone else you'll get your own product not to interact with *new" tools too.

E.g. you produce valid .wat files, but my software which also outputs those has some bits screwed up.

My program can read both .wat but yours can't, but I have 5% market share.

Your users complain they sometimes receive files your software can't read while the competitor can. Do you tell them "well that file is invalid, tell whoever sent it to you to change the software they use"?

The genie can't stay in the bottle unless you have some sort of certification authority and even that may not be enough (see USB)

Re:You can't just assume UTF-8 by thfuran on Tuesday April 30, 2024 @09:26AM (#40210746)
But the more every actor aims to follow the spec rather than reproduce everyone else's bugs, the less quickly the ecosystem devolves into a horrid tangled mess that's nigh impossible for a new entrant.
Re:You can't just assume UTF-8 by quectophoton on Tuesday April 30, 2024 @03:39PM (#40215345)
"So you're saying that I can have an advantage over everyone else if I implement all the spec features, plus one extra feature that adds convenience to our users?

And you're saying that by doing this, not only do I gain an advantage over the existing competition, but I also make it more difficult for more competitors to appear?

[takes notes]"

Re:You can't just assume UTF-8 by thfuran on Wednesday May 01, 2024 @11:21AM (#40224560)
Microsoft is a few decades ahead of you there.
Re:You can't just assume UTF-8 by jimjimjim on Monday April 29, 2024 @09:42PM (#40206398)
This is the answer.

The customer does not know. They just want it to work. They may be using something that someone else gave them. The original source system of the file may not be changeable. But most importantly, their boss just wants it to work. or else.

Re:You can't just assume UTF-8 by tedunangst on Monday April 29, 2024 @07:37PM (#40205508)
Seems like you've uploaded a jpeg. Let me OCR that into CSV for you. Hmm, no text found. Let's pass it to a multimodal LLM.
Re:You can't just assume UTF-8 by arp242 on Monday April 29, 2024 @07:43PM (#40205560)
> What ever happened to the Robustness Principle

Bush hid the facts

Re:You can't just assume UTF-8 by cesarb on Tuesday April 30, 2024 @10:16AM (#40211171)
> Bush hid the facts

For those who don't know that reference: https://en.wikipedia.org/wiki/Bush_hid_the_facts

(A text file containing only the ASCII bytes "Bush hid the facts", when opened in Windows Notepad, displays a sequence of CJK characters instead of the expected English sentence.)

Re:You can't just assume UTF-8 by david422 on Monday April 29, 2024 @08:35PM (#40206004)
> "deals with it,"

This basically loses data integrity if it's wrong though.

You might want to do that with human input if it's helpful to the user - ie user enters a phone number and you strip dashes etc. But if it's machine to machine, it should just follow the spec.

Re:You can't just assume UTF-8 by kazinator on Monday April 29, 2024 @08:47PM (#40206078)
What happened to it:

https://en.wikipedia.org/wiki/Robustness_principle#Criticism

Postel's Law doesn't pass a software engineering smell test.

The idea that software should guess and repair bad inputs is deeply flawed. It is a security threat and a source of enshittification.

Re:You can't just assume UTF-8 by chuckadams on Monday April 29, 2024 @10:02PM (#40206501)
> enshittification

I am so over this word.

Re:You can't just assume UTF-8 by kazinator on Tuesday April 30, 2024 @02:13AM (#40207735)
I put a TODO item in my calendar for December 2024 to stop using it. :)
Re:You can't just assume UTF-8 by ykonstant on Tuesday April 30, 2024 @07:08AM (#40209652)
The enshittification of enshittification was an inevitability.
Re:You can't just assume UTF-8 by jppittma on Monday April 29, 2024 @09:22PM (#40206275)
I think people have collectively decided that they want their programs stupid and predictable, rather than smart and unwieldy.
Re:You can't just assume UTF-8 by vidarh on Monday April 29, 2024 @10:21PM (#40206608)
That is fine in contexts where a wrong guess does no harm.

But that is not always the case, and e.g. silently "fixing" text encoding issues can often corrupt the data if you get it wrong.

By all means offer options of you want, but if you do flag very clearly to the user that they're taking a risk of corrupting the data unless any errors are very apparent and trivial to undo.

Re:You can't just assume UTF-8 by lamontcg on Tuesday April 30, 2024 @12:36AM (#40207287)
I've lived through dealing with non-UTF8 encoding issues and it was a truly gigantic pain in the ass. I'm much more on the side now of people who only want to deal with UTF8 and fully support software that tells any other encoding to go pound sand. The harder life gets for people who use other encodings (yes, particularly Microsoft) the more incentive they have to eventually get on board and stop costing everyone time and effort managing all this nonsense.
Re:You can't just assume UTF-8 by kelnos on Tuesday April 30, 2024 @01:22AM (#40207503)
The article addresses this, that current thinking in many places is that the robustness principle / Postel's Law maybe wasn't the best idea.

If you reject malformed input, then the person who created it has to go back and fix it and try again. If you interpret malformed input the best you can (and get it right), then everyone else implementing the same thing in the future now also has to implement your heuristics and workarounds. The malformed input effectively becomes a part of the spec.

This is why HTML & CSS are the garbage dump they are today, and why different browsers still don't always display pages exactly alike. The reason HTML5 exists is because people finally just gave up and decided to standardize all the broken behavior that was floating around in the wild. Pre-HTML5, the web was an outright dumpster fire of browser compatibility issues (as opposed to the mere garbage dump we have today).

Anyway, it's not really important to try to convince you that Postel's Law is bad; what's important is that you know that many people are starting to think it's bad, and there's no longer any strong consensus that it was ever a good thing.