News for hackers. Stuff that matters.

A comment on A popular but wrong way to convert a string to uppercase or lowercase. Parent comment

Re:A popular but wrong way to convert a string to uppercase ... by zahlman on Tuesday October 08, 2024 @02:46PM (#41780453)
Python's strings have uppercase, lowercase and case-folding methods that don't choke on this. They don't use UTF-16 internally (they can use UCS-2 for strings whose code points will fit in that range; while a string might store code points from the surrogate-pair range, they're never interpreted as surrogate pairs, but instead as an error encoding so that e.g. invalid UTF-8 can be round-tripped) so they're never worried about surrogate pairs, and it knows a few things about localized text casing:

    >>> 'ß'.upper()
    'SS'
    >>> 'ß'.lower()
    'ß'
    >>> 'ß'.casefold()
    'ss'
There are a lot of really complicated tasks for Unicode strings. String casing isn't really one of them.

(No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)

[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by tedunangst on Tuesday October 08, 2024 @03:18PM (#41780786)
But that's wrong. The upper case for ß is ẞ.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by IncreasePosts on Tuesday October 08, 2024 @04:15PM (#41781403)
That was only adopted in Germany like 7 years ago!
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by kccqzy on Tuesday October 08, 2024 @05:10PM (#41781860)
Well languages and conventions change. The € sign was added not that long ago and it was somewhat painful. The Chinese language uses a single character to refer to chemical elements so when IUPAC names new elements they will invent new characters. Etc.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by extraduder_ire on Tuesday October 08, 2024 @09:10PM (#41783486)
Does unicode have space set aside for those new symbols to slot into? I know it's very rare, but it could get messy.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:16AM (#41787646)
Unicode is already messy. Chinese characters especially so due to han unificiation.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by cm2187 on Tuesday October 08, 2024 @04:27PM (#41781531)
C#'s "ToUpper" takes an optional CultureInfo argument if you want to play around with how to treat different languages. Again, solved problem decades ago.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:14AM (#41787625)
This is not a locale issue, it's a Unicode version issue. Which hightlights another problem with adding this to the base standard library.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by Towaway69 on Wednesday October 09, 2024 @02:29AM (#41785090)
Isn't uppercase for ß just ß - i.e. it's its own uppercase character?
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by bratwurst3000 on Wednesday October 09, 2024 @08:59AM (#41787451)
there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps. please correct me if I am wrong. If written in uppercase it should be converted to SZ or the new uppercase ß…. which my iphone doesn’t have… and converting anything to uppercase SS isn’t something germany wants …
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:22AM (#41787713)
> there shouldn’t be an uppercase version of ß because there is no word in the german language that uses it as the first letter. the german language didnt think of allcaps.

Allcaps (and smallcaps) has always existed in signage everywhere. Before the computing age, letters where just arbitrary metal stamps -- and just whatever you could draw before that. Historically, language was not as standardized as it is today.

[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by Towaway69 on Wednesday October 09, 2024 @05:47PM (#41793036)
I don’t think that Germany wants a capital ß or the German language requires one rather technology needs one to dot the eyes and cross the tees.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by account42 on Wednesday October 09, 2024 @09:18AM (#41787669)
Not generally no, but some applications used it that way because of ambiguity of upppercasing ß to SS - which is why ẞ was added.
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by Towaway69 on Wednesday October 09, 2024 @05:43PM (#41793004)
On the other hand, the German language has existed for several hundred years without having a capital ß but now it needs one?

True capitalisation has always existed but even that didn’t seem to have required a capital ß - why now?

[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by kccqzy on Tuesday October 08, 2024 @03:04PM (#41780647)
Still breaks on, for example, Turkish i vs İ. It's impossible to do correctly without language information.

> (No, Python can't turn 'SS' back into 'ß'. But doing that requires metadata about language that a string simply doesn't represent.)

Yes that's my point. Because in typical languages strings don't store language metadata, this is impossible to do correctly in general.

[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by zahlman on Tuesday October 08, 2024 @03:08PM (#41780685)
I'm not seeing anything in the Swift documentation about strings carrying language metadata, either, though?
[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by kccqzy on Tuesday October 08, 2024 @03:20PM (#41780805)
This lowercase function takes a locale argument https://developer.apple.com/documentation/foundation/nsstrin...

It looks like an old NSString method that's available in both Obj-C and Swift.

The casefold function is even older than that. https://developer.apple.com/documentation/foundation/nsstrin... Its documentation specifically includes a discussion of the Turkish İ/I issue.

[ Parent | on HN ]
Re:A popular but wrong way to convert a string to uppercase ... by crote on Wednesday October 09, 2024 @04:07AM (#41785596)
But that's wrong. The uppercase for "in Maßen" ("in moderate amounts") is not "IN MASSEN" ("in Massen", meaning "in massive amounts").
[ Parent | on HN ]