News for hackers. Stuff that matters.
Re:A popular but wrong way to convert a string to uppercase ... by vardump on Tuesday October 08, 2024 @05:12AM (#41775255)
> as there is almost no language that can be written using just that.

99% of use cases I've seen have nothing to do with human language.

1% human language case that is needs to be handled properly using a proper Unicode library.

Your mileage (percentages) may vary depending on your job.

Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Tuesday October 08, 2024 @06:06AM (#41775546)
Right. That’s why I still get mail with my name mangled and my street name barely recognisable. Because I’m in the 1%. Too bad for me…

In all seriousness, though, in the real world ASCII works only for a subset of a handful of languages. The vast majority of the population does not read or write any English in their day to day lives. As far as end users are concerned, you should probably swap your percentages.

ASCII is mostly fine within your programs like the parser you mention in your other comment. But even then, it’s better if a Chinese user name does not break your reporting or logging systems or your parser, so it’s still a good idea to take Unicode seriously. Otherwise, anything that comes from a user or gets out of the program needs to behave.

Re:A popular but wrong way to convert a string to uppercase ... by vardump on Tuesday October 08, 2024 @06:12AM (#41775600)
I said use a Unicode library if input data is actual human language. Which names and addresses are.

99% case being ASCII data generated by other software of unknown provenance. (Or sometimes by humans, but it's still data for machines, not for humans.)

Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Tuesday October 08, 2024 @09:04AM (#41776927)
I am really not sure about this 99%. A lot of programs deal with quite a lot of user-provided data, which you don’t control.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @10:56AM (#41788581)
User-provided data, yes, but also data where you can treat non-ASCII bytes as garbage in -> garbage out. E.g. the config file might be typed by a human but if you need to support case-insensitive keys you still don't need to worry about Unicode.
Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Thursday October 10, 2024 @06:08PM (#41804076)
Exactly. But in this case, don’t try to upper-case or otherwise transform anything.
Re:A popular but wrong way to convert a string to uppercase ... by Muromec on Tuesday October 08, 2024 @06:14AM (#41775623)
Who and why still tries to lowercase/uppercase names? Please tell them to stop.
Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Tuesday October 08, 2024 @09:05AM (#41776932)
Hell if I know. I don’t know what kind of abomination e-commerce websites run on their backend, I just see the consequences.
Re:A popular but wrong way to convert a string to uppercase ... by numpad0 on Tuesday October 08, 2024 @08:34PM (#41783312)
> That’s why I still get mail with my name mangled

Which is why you always type out addresses in ASCII representations in any foreign transactions even if it's not going to match your identity documents, unless the other party specifically demands it in UTF-8 and insists that they can handle it.

> it’s better if a Chinese user name does not break your reporting or logging systems

You should not be just casually dumping Chinese usernames into logs without warnings, in fact, you should not be using Chinese characters for usernames at all. Lots of Chinese online services exclusively use numeric IDs and e-mails for login IDs. "Usernames in natural human language" is a valid concept only in ASCII cultural sphere.

Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Thursday October 10, 2024 @06:07PM (#41804072)
> Which is why you always type out addresses in ASCII representations in any foreign transactions even if it's not going to match your identity documents, unless the other party specifically demands it in UTF-8 and insists that they can handle it.

That is not always possible and the translation from local writing system to ASCII is often not unique and ambiguous. There really is no excuse for this sort of thinking. Even American programmers have to realise at some point that programs serve some purpose and that their failure to represent how the world works is just that: a failure. There is no excuse for programs to not support UTF-8 from user input to any output, including all the processing in between.

Re:A popular but wrong way to convert a string to uppercase ... by Factory on Tuesday October 08, 2024 @09:00PM (#41783435)
"The vast majority of the population does not read or write any English in their day to day lives." This is doubtful: https://en.wikipedia.org/wiki/List_of_languages_by_total_num... While English speakers are not a majority, it is the most popular language. And one should also note that given English is the lingua franca of programming, I'd suspect that English as a second language is actually a majority for programmers. So any code that deals solely with programmers as users can easily just use standard ASCII as default, and never see any problems.
Re:A popular but wrong way to convert a string to uppercase ... by kergonath on Thursday October 10, 2024 @06:03PM (#41804050)
> "The vast majority of the population does not read or write any English in their day to day lives." This is doubtful: https://en.wikipedia.org/wiki/List_of_languages_by_total_num... While English speakers are not a majority, it is the most popular language.

That is the number of English-speaking people, as in people who can speak English. Not necessarily people who use it every day. In any case, ASCII only works for a subset of even English if you ignore all loan words and diacritics in things like proper names.

> So any code that deals solely with programmers as users can easily just use standard ASCII as default, and never see any problems.

That would not be much code at all, given that most code deals with user interfaces or user-provided data. That is the point: it’s not because the code is in basic English simplified enough to fit in ASCII that you can ignore Unicode and don’t need to consider text encoding.

Re:A popular but wrong way to convert a string to uppercase ... by 9dev on Tuesday October 08, 2024 @05:21AM (#41775310)
It's funny how software developers live in bubbles so much. Whether you deal with human language a lot or almost not at all depends entirely on your specific domain. Anyone working on user interfaces of any kind must accommodate for proper encoding, for example; that includes pretty much every line-of-business app out there, which is a lot of code.
Re:A popular but wrong way to convert a string to uppercase ... by elpocko on Tuesday October 08, 2024 @06:32AM (#41775771)
Every search feature everywhere has to be case-insensitive or it's unusable. Search seems like a pretty ubiquitous feature in a lot of software, and has to work regardless of locale/encoding.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @10:58AM (#41788601)
Search needs a whole lot more normalization than just case folding.
Re:A popular but wrong way to convert a string to uppercase ... by elpocko on Wednesday October 09, 2024 @06:24PM (#41793390)
Okay.
Re:A popular but wrong way to convert a string to uppercase ... by inexcf on Tuesday October 08, 2024 @05:41AM (#41775402)
Why do you need upper- or lowercase conversion in cases that have nothing to do with human language?
Re:A popular but wrong way to convert a string to uppercase ... by vardump on Tuesday October 08, 2024 @05:51AM (#41775451)
Here's an example. Hypothetically say you want to build an HTML parser.

You might encounter tags like <html>, <HTML>, <Html>, etc., but you want to perform a hash table lookup.

So first you're going to normalize to either lower- or uppercase.

Re:A popular but wrong way to convert a string to uppercase ... by Muromec on Tuesday October 08, 2024 @06:16AM (#41775642)
But but, I want to have a custom web component and register it under my own name, which can only be properly written in Ukrainian Cyrillic. How dare you not let me have it.
Re:A popular but wrong way to convert a string to uppercase ... by inexcf on Tuesday October 08, 2024 @08:30AM (#41776614)
Ah, i see, we disagree on what is "human language". An abbreviation like HTML and it's different capitalisations to me sound a lot like a feature of human language.
Re:A popular but wrong way to convert a string to uppercase ... by recursive on Tuesday October 08, 2024 @02:24PM (#41780264)
Is this a serious argument? Humans don't directly use HTML to communicate with each other. It's a document markup language rendered by user agents, developed against a specification.
Re:A popular but wrong way to convert a string to uppercase ... by tannhaeuser on Tuesday October 08, 2024 @05:04PM (#41781806)
Markup languages and SGML in particular absolutely are designed for digital text communication by humans and to be written using plain text editors; it's kindof the entire point of avoiding binary data constructs.

And to GP, SGML/HTML actually has a facility to define uppercasing rules beyond ASCII, namely the LCNMSTRT, UCNMSTRT, LCNMCHAR, UCNMCHAR options in the SYNTAX NAMING section in the SGML declaration introduced in the "Extended Naming Rules" revision of ISO 8879 (SGML std, cf. https://sgmljs.net/docs/sgmlrefman.html). Like basically everything else on this level, these rules are still used by HTML 5 to this date, and in particular, that while elements names can contain arbitrary characters, only those in the IRV (ASCII) get case-folded for canonization.

Re:A popular but wrong way to convert a string to uppercase ... by recursive on Tuesday October 08, 2024 @07:58PM (#41783073)
HTML is a text-based medium. But that doesn't make it a human language. Some human languages are not text-based. And some text is not a human language.

ANSI C was designed to be written by humans using a plain text editor. That doesn't make it a human language.

Re:A popular but wrong way to convert a string to uppercase ... by ARandumGuy on Tuesday October 08, 2024 @02:45PM (#41780449)
Converting string case is almost never something you want to do for text that's displayed to the end user, but there are many situations where you need to do it internally. Generally when the spec is case insensitive, but you still need to verify or organize things using string comparison.