News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by thaumasiotes on Tuesday April 30, 2024 @01:16AM (#40207470)
> We don't go "oh that input that's supposed to be json? It looks like a malformed csv file, let's silently have a go at fixing that up for you". Or at least we shouldn't, some software probably does.

Browsers used to have a menu option to choose the encoding you wanted to use to decode the page.

In Firefox, that's been replaced by the magic option "Repair Text Encoding". There is no justification for this.

They seem to be in the process of disabling that option too:

> Note: On most modern pages, the Repair Text Encoding menu item will be greyed-out because character encoding changes are not supported.

( https://support.mozilla.org/en-US/kb/text-encoding-no-longer... )

This note is logical gibberish; encoding isn't something that has to be supported by the page. Decoding is a choice by the browser!

Re:You can't just assume UTF-8 by shiomiru on Tuesday April 30, 2024 @03:17AM (#40208116)
It seems the decision was made in the name of security:

https://hsivonen.fi/no-encoding-menu/

> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden). With the current approach, the parser needs to know of one flag to force chardetng, which the parser has to be able to run in other situations anyway, to run.

> Elaborate UI surface for a niche feature risks the whole feature getting removed

> Telemetry [...] suggested that users aren’t that good at choosing correctly manually.

In other words, it's trying to protect users from themselves by dumbing down the browser. (Never mind that people who know what they are doing have probably also turned off telemetry...)

Re:You can't just assume UTF-8 by thaumasiotes on Tuesday April 30, 2024 @05:06AM (#40208762)
Wow. That writeup is insane.

>> Supporting the specific manually-selectable encodings caused significant complexity in the HTML parser when trying to support the feature securely (i.e. not allowing certain encodings to be overridden).

There's no explanation of why you'd want this, or why it's security-relevant.

(Farther down, there's a mention of self-XSS, which definitely isn't relevant.)

>> Elaborate UI surface for a niche feature risks the whole feature getting removed

They've already removed the whole feature. That was easier to do after they mostly disabled it, not harder.

>> Telemetry showed users making a selection from the menu when the encoding of the page being overridden had come from a previous selection from the menu.

That would be an example of "working as expected". The removal of the ability to do this is the problem that disabling the encoding menu causes! Under the old, correct approach, you'd guess what the encoding was until you got it right. Under the new approach, the browser guesses for you, and if the first guess is wrong, screw you.

Re:You can't just assume UTF-8 by n_plus_1_acc on Tuesday April 30, 2024 @03:13AM (#40208092)
Probably because most websites now send a correct encoding header or meta Tag, so the user changing can only make it wrong. (Assuming no encoding header is wrong, which happens in reality)
Re:You can't just assume UTF-8 by layer8 on Tuesday April 30, 2024 @07:39AM (#40209884)
It does happen a lot that old text content in non-UTF-8 encoding is mistakenly served explicitly marked as UTF-8. It is precisely in such circumstances that the Repair Text option is useful.