News for hackers. Stuff that matters.
Re:TIL the assumption that string length does not change whe... by Dylan16807 on Monday November 22, 2021 @04:26AM (#29304497)
Except for SMS, #2 should work fine for all of those uses. And the reason you'd need a different measure for SMS is because SMS is bad.

And half the things on your list are just places that Unicode itself might fail, regardless of what the length is? That has nothing to do with how you count, because the number you get won't matter.

Re:TIL the assumption that string length does not change whe... by jasode on Monday November 22, 2021 @06:30AM (#29305265)
>, #2 should work fine for all of those uses.

No, #2-length is a higher-level of abstraction to display for humans. E.g. calculating pixel widths for sizing a column in a GUI table, etc.

I don't think you understood the lower-level of abstraction in the examples that gp (dotancohen) laid out. (Encrypting/compressing/transmitting/etc strings.) In those cases, you have to know exact count of bytes and not the composed graphemes that collapse to a human visible "length".

In other words, the following example code to allocate a text buffer array would be wrong:

  char *t = malloc(length_type_2); // buggy code with an undersized buffer
Re:TIL the assumption that string length does not change whe... by zionic on Monday November 22, 2021 @09:58AM (#29306929)
>is a higher-level of abstraction to display for humans

Am I missing something here? Who do you think is actually writing this stuff? Aliens?

I just want the first 5 characters of an input string, or to split a string by ":" or something.

I have plenty of apps where none of my strings _ever_ touch a database and are purely used for UI. I'll ask this again: Why break strings _globally_ to solve some arcane <0.0001% edge case instead of fixing the interface at the edges for that use case?

All this mindset has done is cause countless hours of harm/frustration in the developer community.

Re:TIL the assumption that string length does not change whe... by int_19h on Monday November 22, 2021 @04:06PM (#29311568)
It's causing frustration because it's breaking people's preconceived notions about how text is simple. Previously, the frustration caused was usually to the end users of apps written by people with those preconceived notion. And there are far more end users than developers, so...
Re:TIL the assumption that string length does not change whe... by jasode on Monday November 22, 2021 @11:56AM (#29308565)
>Am I missing something here? [...] I have plenty of apps where none of my strings _ever_ touch a database and are purely used for UI.

Yes, the part you were missing is that I was continuing a discussion about a specific technical point brought up by parents' Dylan16807 and dotancohen. The fact that you manipulate a lot strings without databases or persistence/transmission across other i/o boundaries is not relevant to the context of my reply. To restate that context... Dylan16807's claim that length_type_#2 (counting human visible grapheme clusters) is all the information needed for i/o boundaries is incorrect and will lead to buggy code.

With that aside, I do understand your complaint in the following:

>I just want the first 5 characters of an input string, or to split a string by ":" or something.

This is a separate issue about ergonomics of syntax and has been debated often. A previous subthread from 6 years ago had the same debate: https://news.ycombinator.com/item?id=10519999

In that thread, some commenters (tigeba, cookiecaper) has the same complaint as you that Swift makes it hard to do simple tasks with strings (e.g. IndexOf()). The other commenters (pilif, mikeash) respond that Swift makes the high-level & low-level abstractions of Unicode more explicit. Similar philosophy in this Quora answer by a Swift compiler contributor highlighting tradeoffs of programmers not being aware if string tasks are O(1) fast -vs- O(n) slow:

https://www.quora.com/Why-is-string-manipulation-so-difficul...

And yes your complaint is shared by many... By making string api explicitly confront the "graphemes -vs- codeunits - codepoints -vs- bytes", you do get clunky cumbersome syntax such as "string.substringFromIndex(string.startIndex.advancedBy(1))" as in the Q&A answer:

https://stackoverflow.com/questions/2503436/how-to-check-if-...

I suppose Swift could have designed the string api to your philosophy. I assume the language designers anticipated it would have lead to more bugs. The tradeoff was:

- "cumbersome syntax with less bugs"

...or...

- "easier traditional syntax (like simple days of ASCII) with more hidden bugs and/or naive performance gotchas"

Re:TIL the assumption that string length does not change whe... by Dylan16807 on Monday November 22, 2021 @05:39PM (#29312431)
Manually allocating a buffer wasn't on the list of things I replied to, and you shouldn't be manually allocating buffers that way in the first place.
Re:TIL the assumption that string length does not change whe... by CodesInChaos on Monday November 22, 2021 @05:45AM (#29304995)
The main reason I care about the length of strings is to limit storage/memory/transmission size. 1) and to a lesser degree 4) achieve that (max 4 utf8 bytes per codepoint).

2) comes with a log of complexity, so I'd only use 2) in a few places where string lengths need to be meaningful for humans and when stepping through a string in the UI.

* One (extended) grapheme cluster can require a lot of storage (I'm not sure if it's even bounded at all), so it's unsuitable for length limitations

* Needs knowledge of big and regularly updated unicode tables. So if you use it in an API chances are high that both sides will interpret it differently, so it's unsuitable for API use.

Re:TIL the assumption that string length does not change whe... by Dylan16807 on Monday November 22, 2021 @05:45PM (#29312494)
> One (extended) grapheme cluster can require a lot of storage (I'm not sure if it's even bounded at all), so it's unsuitable for length limitations

You could take inspiration from one of the annexes and count no more than 31 code points together.

> Needs knowledge of big and regularly updated unicode tables. So if you use it in an API chances are high that both sides will interpret it differently, so it's unsuitable for API use.

It depends on what you're doing. That could cause problems, but so could counting bytes. Users will not be thrilled when their character limit varies wildly based on which characters they use. Computers aren't going to be thrilled either when replacing one character with another changes the length. Or, you know, when they can't uppercase a string.