News for hackers. Stuff that matters.
Re:A popular but wrong way to convert a string to uppercase ... by SleepyMyroslav on Tuesday October 08, 2024 @01:59PM (#41780014)
In gamedev there is simple rule: don't try to do any of that.

If it is text game needs to show to user then every version of the text that is needed is a translated text. Programmer will never know if context or locale will need word order changes or anything complicated. Just trust the translation team.

If text is coming from user - then change design until its not needed to 'convert'. There are major issues just to show user back what he entered! Because the font for editing and displayed text could be different. Not even mentioning RTL and other issues.

Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :)

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @02:58PM (#41780568)
>Once ppl learn about localization the questions like why a programming language does not do this 'simple text operation' are just a newcomer detector. :)

I think you are purposefully misinterpreting the question. They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters.

What if your game needs to talk to a server and do some string manipulation in between requests? Are you really going to architect everything so that the client doesn't need to handle any of that ever?

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Tuesday October 08, 2024 @03:48PM (#41781113)
> They're not asking about converting the case of any Unicode string with locale sensitivity, they're asking about converting the case of ASCII characters.

I'm confused now. The article specifically mentions issues with UTF-16 and UTF-32 unicode characters outside the basic multilingual plane (BMP).

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @04:00PM (#41781244)
I'm referring to the people who call case conversion in general "a simple text operation". Say you have an std::string and you want to make it lower case. If you assume it contains just ASCII that's a simpler operation than if you assume it contains UTF-8, but C++ doesn't provide a single function that does either of them. A person can rightly complain that the former is a basic functionality that the language should include; personally, I would agree. And you could say "wow, doesn't this person realize that case conversion in Unicode is actually complicated? They must be really inexperienced." It could be that the other person really doesn't know about Unicode, or it could mean that you and them are thinking about entirely different problems and you're being judgemental a bit too eagerly.
Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Tuesday October 08, 2024 @04:18PM (#41781440)
For ascii in C++ isn't there std::tolower / std::toupper? If you're not dealing with unsigned char types there isn't a simple case conversion function, but that's for a good reason as the article lays out.
Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @04:45PM (#41781662)
Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple case conversion functions that operate on strings in-place.
Re:A popular but wrong way to convert a string to uppercase ... by theelous3 on Tuesday October 08, 2024 @07:24PM (#41782846)
> There's really no excuse for the library not having simple case conversion functions that operate on strings in-place.

Could not agree more. Any time I touch a C I want to scoop my brain out of my ear. So many simple unbelievably common operations have fifty "best" ways to do them, when they should have one happy path 99% of usecases require baked in. Nobody should ever have to seriously consider something as ridiculous as "is tolower addressable?".

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Tuesday October 08, 2024 @08:32PM (#41783304)
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside.

The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place.

If you are operating on wide strings, there is no suitable single solution, partly because wstring is a terrible type. It's different widths on different platforms, and no string encoding format uses a generalized wsring, they have mandatory min/max character byte widths. So a wstring tells you nothing about the actual encoded string contents semantic representation.

The C++ stdlib could include a fully unicode aware string type set, and surrounding library. But personally I think C++ isn't the kind of language to provide an opinionated stdlib module for such a complex task. And there's no way to implement such a module without being very opinionated about something.

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @09:35PM (#41783616)
I know I can simply iterate. The point is that it's a function that should be included, not that it's impossible without it. It's one of the most common string operations.
Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @08:48AM (#41787346)
To me that feels like the JS community asking for left-pad or is-even in a module. Why have a dedicated function for 2 lines of code?

And it's a huge footgun. There is no ascii type in C++. People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.

You could say the generalized tolower should support all the different width/encoding combinations and sort it out. But that's still highly opinionated as far as performance is concerned.

Generalized string conversion is a very complex problem and you really cannot simplify it in a way that will satisfy most C++ users. Just use ICU or utf8cpp if you want to do string operations and don't care what's going on under the hood. But even then I can't recommend just 1 library, because no perfect 3rd party library exists. A perfect first party library definitely could not exist.

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Wednesday October 09, 2024 @09:51AM (#41787974)
>Why have a dedicated function for 2 lines of code?

Then why does std::max() exist?

>People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.

tolower() and toupper() work correctly on UTF-8 strings, because UTF-8 was specifically designed so that non-ASCII characters were represented by sequences of purely non-ASCII bytes.

>Generalized string conversion is a very complex

Hence why people who say C++ should have a tolower() that operates on strings are not asking more complex Unicode support.

Re:A popular but wrong way to convert a string to uppercase ... by dwattttt on Wednesday October 09, 2024 @05:10AM (#41785896)
> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place.

Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant).

7-bit ASCII code points are fine, but outside that it's not guaranteed that one UTF-8 byte converts into exactly one UTF-8 byte when converting case.

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @08:40AM (#41787266)
Yeah If you're using narrow strings for UTF8 you're making a mistake. wstrings also are not a good representation because of the platform differences, unless you don't care about Windows in which case it's fine but still not great semantically.

In most type definitions you cannot convert UTF8 via simple iteration because the type generally represents a code point and not a character.

You can have a library where UTF8 characters are a native type and code points are a mostly-hidden internal element. But again, that's highly opinionated for C++.

Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Thursday October 10, 2024 @05:27AM (#41797113)
I'm not 100% sure what you mean by narrow string, but if you refer to std::string vs std::wstring, then std::string is perfectly fine for encoding UTF8, as that uses 8 bit code units which are guaranteed to fit in a char. On the other hand, std::wstring would be a bizarre choice for UTF8 on any platform.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @08:44AM (#41787314)
It's not guaranteed for 7-bit ASCII either because tolower/toupper are locale-dependent and with the tr_TR lowercase I (U+0049) is ı (U+0131, aka dotless i) wich encodes as two bytes in UTF-8.
Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @09:02AM (#41787482)
That's not ascii then. It's byte width compatible (to a certain degree as you point out). But it's not ascii. ascii defines 128 code points and the handling of an escape character. It doesn't handle locales.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:29AM (#41787765)
ASCII is an encoding, it doesn't say anything about locale. The point is that tolower/toupper is not guaranteed to be safe even if the input is 7-bit.
Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Thursday October 10, 2024 @05:16AM (#41797062)
I don't think there is any possibility of doing locale specific lower/upper casing in ASCII. It is really designed for (a subset of) American english.
Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Wednesday October 09, 2024 @08:52AM (#41787387)
std::u8string, std::u16string and std::u32string are supposed to be the portable unicode string types, but a lot of machinery is missing and some that has been added has since been deprecated.

> there's no way to implement such a module without being very opinionated about something.

indeed! Boost.Nowide[1] is such an opinionated library.

[1] https://www.boost.org/doc/libs/master/libs/nowide/doc/html/i...

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @09:04AM (#41787509)
Yep, there's also ICU and utf8cpp, and many others. They all have trade-offs. So I just don't think the stdlib should cover this because there is no objectively best way to handle it.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @08:40AM (#41787267)
std::tolower / std::toupper are rubbish functions that can't do proper Unicode but still pull in the bloated locale machinery for what should be a simple conditional integer addition if all you care about is ASCII. Both have no valid use case and should be marked [[deprecated]] and erased from all teaching materials.
Re:A popular but wrong way to convert a string to uppercase ... by SleepyMyroslav on Tuesday October 08, 2024 @06:30PM (#41782449)
>What if your game needs to talk to a server and do some string manipulation in between requests? Are you really going to architect everything so that the client doesn't need to handle any of that ever?

Of course! Your string manipulation with user entered attributes like display names or chat messages are 1 millimeter away from old good sql 'Bobby; drop table students'. Never ever do that if you can avoid it. Every time someone 'just concatenates' two strings like to add ie 'symbol that represents input button' programmer makes bad bug that will be both annoying and wrong. Games should use substitution patterns guided by translation team. Because there is no ASCII culture in like around 15 typically supported by big publishers.

There are exceptions like platform provided services to filter ban words in chat. And even there you don't have to do 'things with ASCII characters'. Yeah, players will input unsupported symbols everywhere they can and you need to have good replacement characters for those and fix support for popular emojis regularly. That is expected by communities now.

Re:A popular but wrong way to convert a string to uppercase ... by lmm on Tuesday October 08, 2024 @08:52PM (#41783402)
> What if your game needs to talk to a server and do some string manipulation in between requests?

What conceivable reason would there be to ever need to do that? If the server takes commands in upper case, then have them in upper case from the start. If the server takes commands in lower case, have them in lower case from the start. If the server specifies that you need to invert the case of its response to use in the next request, find a server developed by someone not crazy.

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @09:13PM (#41783502)
Case conversion is not the only string manipulation that's locale sensitive.
Re:A popular but wrong way to convert a string to uppercase ... by lmm on Tuesday October 08, 2024 @09:39PM (#41783630)
No reasonable server API should require locale sensitive string manipulation.
Re:A popular but wrong way to convert a string to uppercase ... by NBJack on Wednesday October 09, 2024 @12:36AM (#41784533)
Sorry, I think you misspelled "true" and "Scotsman" in that sentence.

Examples and reasoning for your assertion?

Re:A popular but wrong way to convert a string to uppercase ... by NBJack on Wednesday October 09, 2024 @12:33AM (#41784524)
Word censoring? Ease of use? Console commands (i.e. from Quake to minecraft)?
Re:A popular but wrong way to convert a string to uppercase ... by lmm on Wednesday October 09, 2024 @02:44AM (#41785163)
> Word censoring?

Should only ever be needed for text from the user, and in that case, as GP said, find a way to examine it as-is, don't "convert".

> Ease of use?

What ease of use? When has futzing around with case ever made anything easier?

> Console commands (i.e. from Quake to minecraft)?

Why would those necessitate changing case?

Re:A popular but wrong way to convert a string to uppercase ... by wongarsu on Wednesday October 09, 2024 @12:49AM (#41784604)
Those sound exactly like the newcomer detectors GP was referring to. What you want is a case-insensitive string comparison, and outside ASCII that's not equivalent to just turning both strings to lowercase and checking equality (or doing a substring search or whatever the task requires)
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @08:48AM (#41787351)
Exactly and where you want case-insensitive comparison you almost always also want other kinds of Unicode normalization.
Re:A popular but wrong way to convert a string to uppercase ... by barrkel on Wednesday October 09, 2024 @03:46AM (#41785479)
Nobody is thinking about converting the case of ASCII characters. To be thinking that, they are explicitly excluding most of the world's cultures from entering common names correctly. Restricting thought to ASCII is a lack of thought, not an active thought.
Re:A popular but wrong way to convert a string to uppercase ... by zahlman on Tuesday October 08, 2024 @02:51PM (#41780508)
>If text is coming from user - then change design until its not needed to 'convert'

In games, you can possibly get away with this. Most other people need to worry about things like string collation (locale-aware sorting) for user-supplied text.

Re:A popular but wrong way to convert a string to uppercase ... by makeitdouble on Tuesday October 08, 2024 @10:48PM (#41784004)
TBF, if you are caring about string collation, you're already at the entrance of the rabbit hole and probably should go down to the deep end anyway.

I'd assume SleepyMyroslav doesn't apply to devs willing to spend weeks at time to handle all the complexity in full.

Re:A popular but wrong way to convert a string to uppercase ... by beeboobaa3 on Tuesday October 08, 2024 @05:24PM (#41781970)
> There are major issues just to show user back what he entered! Because the font for editing and displayed text could be different. Not even mentioning RTL and other issues.

Your web browser is doing it right now as you are reading this comment.

Re:A popular but wrong way to convert a string to uppercase ... by rty32 on Tuesday October 08, 2024 @06:51PM (#41782605)
And web development is not game development? And chances are that games don't ship chromium with them?
Re:A popular but wrong way to convert a string to uppercase ... by moron4hire on Tuesday October 08, 2024 @09:06PM (#41783463)
Actually...

  https://github.com/baikety/uWebKit
  https://zenfulcrum.com/browser/docs/Readme.html
  https://github.com/roydejong/chromium-unity-server
There are a lot more, I just got bored at 3.

And it's not just Unity. Several exist for Unreal as well.

Why? Specifically because 2D layout and text rendering suck so much in game engines. What's ~50MB matter when you're shipping several GB of game assets?

Re:A popular but wrong way to convert a string to uppercase ... by cheema33 on Wednesday October 09, 2024 @01:09AM (#41784676)
> In gamedev there is simple rule: don't try to do any of that.

I am not in gamedev, but I frequently have to develop middleware that takes in user entered data and formats it in a way that will import into a 3rd party system without errors. And that sometimes means changing the case on strings.

In my experience as a developer, this is very very common requirement.

Luckily I am not forced to use a low level language for any of my work. In C# I can simply do this: "hello world".ToUpper();

Re:A popular but wrong way to convert a string to uppercase ... by Smaug123 on Wednesday October 09, 2024 @03:37AM (#41785429)
If you're putting data into a third-party system, you might want `ToUpperInvariant`, not `ToUpper`. (Just checking that you know the difference, because most people don't!)
Re:A popular but wrong way to convert a string to uppercase ... by pjmlp on Wednesday October 09, 2024 @03:44AM (#41785474)
Note that the correct way to do that in C# would be to pass an instance of CultureInfo.
Re:A popular but wrong way to convert a string to uppercase ... by crote on Wednesday October 09, 2024 @04:02AM (#41785566)
The problem is that such third-party requirements are usually wrong.

Two decades ago some developer probably went "Yeah, obviously all names start with capital letters!", not realizing that there are in fact plenty of names which start with a lowercase letter. So they added an input validation test which checks for capitals, which meant everyone feeding that system had to format their data. A whole ecosystem grew around the format of the output of that system, and now you're suddenly rewriting the system and you run into weird and plain wrong capitalization requirements for no technical reason whatsoever.

Alternatively, the same but start with punch cards which predate ASCII and don't distinguish between uppercase and lowercase letters.

> In C# I can simply do this: "hello world".ToUpper()

... which does not work.

Take a look at the German word "straße" (street), for example. Until very recently the "ß" character did not have an uppercase variant, so a ToUpper would convert it to "STRASSE". This is a lossy operation, as the reverse isn't true: the lowercase variant of "KONGRESSSTRASSE" (congress street) is not "kongreßstraße" - it's supposed to be "Kongressstraße".

It can get even worse: the phrase "in Maßen" (in moderate amounts) naively has the uppercase variant "IN MASSEN" - but that means "in huge amounts"! In that case it is probably better to stick to "IN MASZEN".

And then there's Turkish, where the uppercase variant of the letter "i" is of course "İ" rather than "I" - note the dot.

So no, you cannot "simply" use ToUpper() / ToLower(). They might work well enough of basic ASCII for languages like English, but they have a habit of making a mess out of everything else. You're supposed to use CultureInfo.TextInfo.ToUpperCase() and explicitly specify what locale the text is in so that it can use the right converter. Which is of course essentially impossible in general-purpose text fields.

In practice that means your options are a) giving up on the concept of uppercase/lowercase conversion and just passing it as-is, or b) accepting that you are inevitably going to be silently corrupting your data.

Re:A popular but wrong way to convert a string to uppercase ... by wruza on Wednesday October 09, 2024 @06:34AM (#41786314)
It’s a lossy operation, but it does work. By this logic jpeg and mpeg don’t work either. But were watching them videos daily.

Yes we can simply ToUpper(). We just can’t ToUpper().ToLower(), but that’s useless cause we have the original string if we need it and fine if we don’t need it.

Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @08:53AM (#41787403)
The point is that what ToUpper does depends on locale AND Unicode version. This for many applications it only appears to work until it will fail spectacularly in production.
Re:A popular but wrong way to convert a string to uppercase ... by neonsunset on Wednesday October 09, 2024 @01:53PM (#41790739)
> So no, you cannot "simply" use ToUpper() / ToLower(). They might work well enough of basic ASCII for languages like English, but they have a habit of making a mess out of everything else. You're supposed to use CultureInfo.TextInfo.ToUpperCase() and explicitly specify what locale the text is in so that it can use the right converter. Which is of course essentially impossible in general-purpose text fields.

Have you ever read the documentation? https://learn.microsoft.com/en-us/dotnet/fundamentals/runtim...

Re:A popular but wrong way to convert a string to uppercase ... by crote on Thursday October 10, 2024 @08:15AM (#41798072)
Yes. Now try applying it to something like this very HN comment section, which is mixing words belonging to different cultures inside a single comment - and in some cases even inside the same word.

Sure, you can now do case conversion for a specific culture, but which one?

Re:A popular but wrong way to convert a string to uppercase ... by Netch on Wednesday October 09, 2024 @11:18AM (#41788811)
> In C# I can simply do this: "hello world".ToUpper();

Hmm still actual: https://www.moserware.com/2008/02/does-your-code-pass-turkey...

Re:A popular but wrong way to convert a string to uppercase ... by neonsunset on Wednesday October 09, 2024 @01:54PM (#41790750)
> 2008

This is completely irrelevant because culture-sensitive case conversion relies on ICU/NLS.

Re:A popular but wrong way to convert a string to uppercase ... by Netch on Friday October 11, 2024 @02:47AM (#41806917)
But at least a programmer shall be aware to call it (whatever API is used).
Re:A popular but wrong way to convert a string to uppercase ... by jameshart on Wednesday October 09, 2024 @05:21AM (#41785957)
I don’t think you can say this is universally known in ‘game dev’. In fact just last week I stumbled using the UI in a game that let me enter a name for something, which it then displayed in uppercase.

Game UI is the place I’d expect to most likely come across horrific abuses of localization precisely because game UI is such a cobbled together layer of hacks on hacks.