News for hackers. Stuff that matters.
Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @04:45PM (#41781662)
Those functions take and return single characters. What's missing is functions that operate on strings. You can use them in combination with std::transform(), but as the article points out, even if you're just dealing with ASCII you can easily do it wrong. I've been using C++ for over 20 years and I didn't know tolower() and toupper() were non-addressable. There's really no excuse for the library not having simple case conversion functions that operate on strings in-place.
Re:A popular but wrong way to convert a string to uppercase ... by theelous3 on Tuesday October 08, 2024 @07:24PM (#41782846)
> There's really no excuse for the library not having simple case conversion functions that operate on strings in-place.

Could not agree more. Any time I touch a C I want to scoop my brain out of my ear. So many simple unbelievably common operations have fifty "best" ways to do them, when they should have one happy path 99% of usecases require baked in. Nobody should ever have to seriously consider something as ridiculous as "is tolower addressable?".

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Tuesday October 08, 2024 @08:32PM (#41783304)
std::transform() seems like overkill when you can just iterate over the string and modify it in place. And in my opinion, tranform is way less readable than seeing a loop over some array with a single operation inside.

The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place.

If you are operating on wide strings, there is no suitable single solution, partly because wstring is a terrible type. It's different widths on different platforms, and no string encoding format uses a generalized wsring, they have mandatory min/max character byte widths. So a wstring tells you nothing about the actual encoded string contents semantic representation.

The C++ stdlib could include a fully unicode aware string type set, and surrounding library. But personally I think C++ isn't the kind of language to provide an opinionated stdlib module for such a complex task. And there's no way to implement such a module without being very opinionated about something.

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Tuesday October 08, 2024 @09:35PM (#41783616)
I know I can simply iterate. The point is that it's a function that should be included, not that it's impossible without it. It's one of the most common string operations.
Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @08:48AM (#41787346)
To me that feels like the JS community asking for left-pad or is-even in a module. Why have a dedicated function for 2 lines of code?

And it's a huge footgun. There is no ascii type in C++. People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.

You could say the generalized tolower should support all the different width/encoding combinations and sort it out. But that's still highly opinionated as far as performance is concerned.

Generalized string conversion is a very complex problem and you really cannot simplify it in a way that will satisfy most C++ users. Just use ICU or utf8cpp if you want to do string operations and don't care what's going on under the hood. But even then I can't recommend just 1 library, because no perfect 3rd party library exists. A perfect first party library definitely could not exist.

Re:A popular but wrong way to convert a string to uppercase ... by fluoridation on Wednesday October 09, 2024 @09:51AM (#41787974)
>Why have a dedicated function for 2 lines of code?

Then why does std::max() exist?

>People will use the generalized tolower for UTF8 encoded in narrow strings and have issues.

tolower() and toupper() work correctly on UTF-8 strings, because UTF-8 was specifically designed so that non-ASCII characters were represented by sequences of purely non-ASCII bytes.

>Generalized string conversion is a very complex

Hence why people who say C++ should have a tolower() that operates on strings are not asking more complex Unicode support.

Re:A popular but wrong way to convert a string to uppercase ... by dwattttt on Wednesday October 09, 2024 @05:10AM (#41785896)
> The article talks about wstrings for good reason. If you're converting narrow strings, you don't need to be this fancy. Just loop over the string and edit it in place.

Since you mention narrow strings in the context of wstring, just to make sure... you can't convert a UTF-8 std::string character by character, in-place (in case that's what you meant).

7-bit ASCII code points are fine, but outside that it's not guaranteed that one UTF-8 byte converts into exactly one UTF-8 byte when converting case.

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @08:40AM (#41787266)
Yeah If you're using narrow strings for UTF8 you're making a mistake. wstrings also are not a good representation because of the platform differences, unless you don't care about Windows in which case it's fine but still not great semantically.

In most type definitions you cannot convert UTF8 via simple iteration because the type generally represents a code point and not a character.

You can have a library where UTF8 characters are a native type and code points are a mostly-hidden internal element. But again, that's highly opinionated for C++.

Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Thursday October 10, 2024 @05:27AM (#41797113)
I'm not 100% sure what you mean by narrow string, but if you refer to std::string vs std::wstring, then std::string is perfectly fine for encoding UTF8, as that uses 8 bit code units which are guaranteed to fit in a char. On the other hand, std::wstring would be a bizarre choice for UTF8 on any platform.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @08:44AM (#41787314)
It's not guaranteed for 7-bit ASCII either because tolower/toupper are locale-dependent and with the tr_TR lowercase I (U+0049) is ı (U+0131, aka dotless i) wich encodes as two bytes in UTF-8.
Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @09:02AM (#41787482)
That's not ascii then. It's byte width compatible (to a certain degree as you point out). But it's not ascii. ascii defines 128 code points and the handling of an escape character. It doesn't handle locales.
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:29AM (#41787765)
ASCII is an encoding, it doesn't say anything about locale. The point is that tolower/toupper is not guaranteed to be safe even if the input is 7-bit.
Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Thursday October 10, 2024 @05:16AM (#41797062)
I don't think there is any possibility of doing locale specific lower/upper casing in ASCII. It is really designed for (a subset of) American english.
Re:A popular but wrong way to convert a string to uppercase ... by gpderetta on Wednesday October 09, 2024 @08:52AM (#41787387)
std::u8string, std::u16string and std::u32string are supposed to be the portable unicode string types, but a lot of machinery is missing and some that has been added has since been deprecated.

> there's no way to implement such a module without being very opinionated about something.

indeed! Boost.Nowide[1] is such an opinionated library.

[1] https://www.boost.org/doc/libs/master/libs/nowide/doc/html/i...

Re:A popular but wrong way to convert a string to uppercase ... by squeaky-clean on Wednesday October 09, 2024 @09:04AM (#41787509)
Yep, there's also ICU and utf8cpp, and many others. They all have trade-offs. So I just don't think the stdlib should cover this because there is no objectively best way to handle it.