News for hackers. Stuff that matters.
Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Monday May 15, 2023 @02:27PM (#35952509)
> This proposal also recommends adoption of Unicode normalization form C (NFC) for identifiers to ensure that when compared, identifiers intended to be the same will compare as equal. Legacy encodings are generally naturally in NFC when converted to Unicode. Most tools will, by default, produce NFC tex.

Er, a much better approach is to allow unnormalized Unicode in source code and use form-insensitive matching of symbol names so that all forms of a symbol are equivalent. This can be done by normalizing during the parse, or by implementing form-insensitive string comparison and hashing functions that normalize glyph by glyph as needed -- the latter can be very fast for all-ASCII and mostly-ASCII symbols!

The reason this is a better way is that there's too many places that don't produce NFC. For example, HFS+ uses NFD, so if you cut-n-paste a file name from HFS+ into other contexts, you'll be pasting NFD unless the cut-n-paste system normalizes to NFC. Also, while it's true that input modes typically produce NFC, it's more that they produce NFC for a small subset of Unicode, not that they will normalize other forms seen on input. Using form-insensitive string comparison/hashing/matching yields a better user experience at not that much implementation cost: you're gonna need a Unicode library, and that library will need to have normalization support, so you can implement form-insensitivity.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by wahern on Monday May 15, 2023 @03:00PM (#35952989)
> Er, a much better approach is to allow unnormalized Unicode in source code and use form-insensitive matching of symbol names so that all forms of a symbol are equivalent.

Linkers will often be blissfully unaware of Unicode or any form of localization. This was the impetus for UTF-8, so that the bulk of software which is 8-bit clean or which operates on opaque, NUL-terminated strings can continue working as-is. This can't be changed without breaking backwards ABI compatibility; therefore, it's very unlikely to change.

There are countless half-measures that could be taken, but few if any are suitable for standardization. If the history of software localization is any guide, in the face of strict, forward-looking specifications various vendors and ecosystems will likely go there own way, with the one sure thing being a failure to fully adopt or properly implement the specification.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Monday May 15, 2023 @06:09PM (#35955185)
Yes, the compiler should normalize symbols before writing object files, no doubt. I'm talking about the inputs though -the source files- which should not have to be normalized.
Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by froh on Tuesday May 16, 2023 @02:21AM (#35958306)
that means the compiler needs to have the Unicode database.

I'd rather have a seperate 'recode' to NFC preprocessing in the build process, if the developers editor doesn't do NFC normalization.

if I'm not mistaken checking for NFC is simpler than transforming to NFC, right?

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Tuesday May 16, 2023 @11:08AM (#35962885)
The compiler is going to need a Unicode library anyways.

It's true that checking that some string is in some canonical form (say, NFC) is easier than normalizing it. In fact, it's even easier to check that some string is in NFD than it is to check that it's in NFC because NFC is defined in terms of NFD. That's not enough to justify not doing form-insensitive string matching in the compiler and forcing a pre-compilation normalization step.

Also, you'd only want symbols normalized, not string literals (since one might need a string literal that is not in a canonical form, or not in NFC). Thus the pre-compilation normalization step would have to be language-aware, so it might as well be part of the compiler and not a separate step.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by froh on Tuesday May 16, 2023 @12:29PM (#35964174)
> you'd only want symbols normalized, not string literals

_this_ is the argument that convinces me.

thank you:-)

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Tuesday May 16, 2023 @12:53PM (#35964541)
You're welcome :)

But I'd like to convince you and everyone that in general we want to accept inputs in any form and be form-insensitive because that's much more user-friendly.

We don't in fact have universal NFC-only input modes. We have accidentally NFC-mostly input modes -- accidental because typically we have transcoding from legacy codesets, which yields NFC, but no normalization is actually done so that copy-paste operations can cause non-canonical input to be provided to applications. We don't have universal agreement on NFC. This makes for a mess with occasional user-visible problems.

Form-insensitivity is essentially the same as normalizing character-by-character when hashing or comparing strings, but this can be very fast for mostly-ASCII text, so form-insensitivity can be very fast.

For string comparison form-insensitivity is faster than first normalizing because either the strings are equal in content and probably equal in form, or they differ early and possibly at some character where normalization is not necessary to determine the result of the comparison, and thus less work need be done to compare strings form-insensitively than first normalizing all inputs (unless so many string comparisons will be done that maybe normalizing first might be a win). For string hashing form-insensitivity is not faster because one has to normalize every input, whereas if all inputs are normalized once then one need never normalize to hash, but still form-insensitivity yields a better user experience.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by rurban on Tuesday May 16, 2023 @02:07PM (#35965518)
Only a tiny part of the Unicode database, the normalization tables. Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc.

And Unicode identifiers are entirely insecure, because they are not identifiable. My proposal was postponed to C26.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Tuesday May 16, 2023 @03:01PM (#35966161)
> Problem is these tables have to be updated every year, and they dont with similar tables, such as the glibc.

The language standard can avoid this by committing to a single Unicode version for each language version.

> And Unicode identifiers are entirely insecure, ...

Eh, it depends on what we're talking about using them for. If it's symbols in object files, it's not really a problem. More importantly confusables are unavoidable -- even ASCII all by itself has confusable characters (1 and l for example). The thing to do is to forbid arbitrary mixing of scripts, allowing only those scripts that are relevant in the relevant contexts. For DNS this means that DNS TLD registries should come up with per-registry rules, for example.

Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by rurban on Wednesday May 17, 2023 @06:09AM (#35973304)
Re:GCC 13 Supports New C2x Features, Including Nullptr, Enha... by cryptonector on Wednesday May 17, 2023 @11:26AM (#35976686)
Can you expand on what the security concerns are as to confusables in symbol names in C source code? Clearly there's a security concern with cut-n-paste, but that's true regardless of what the rules might be for C identifiers.

It's not like it's obvious that UTR#39 applies literally everywhere that there are "identifiers".

Also, can you speak to what is the security concern with form-insensitivity (rather than confusables) as to symbols in input source files? I just don't see a concern at all there, but maybe I'm missing something.

Lastly, I think `#include` is the most important place to get this right since that does interface with the world outside the compiler (specifically: the filesystem), but as you note the filesystems mostly are just-use-8bit -- very few filesystems normalize on create (HFS+) or are form-insensitive (ZFS). The other place to get this right is on the object file output side, where symbols definitely must be normalized.

Oh, one more thing: the platform might impose some rules regarding symbols in ELF and any other object file formats. Are they known to? I suppose C can't necessarily cater to all platform-imposed limitations on symbol naming, but it'd be useful to know about them.