> Should strings be represented as codepoint arrays in memory?

No. I think the implementor of Factor's unicode support originally did that but it turns out to not be useful:

* it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3)

* this also has impact on CPU caches, lowering the average amount of data you can fit in your cache and work on

* it requires a complete conversion of incoming ascii and utf8 data (which only get more and more common as time goes on) rather than just a validation

* and because Unicode itself is variable-length (combining codepoints) it's not actually helpful when you're trying to properly manipulate unicode data

The only "advantage" of representing strings as codepoint arrays is that you get O(1) access to codepoints which is a terrible idea you should not encourage.

UTF-32 internal encoding makes some manipulations very slightly easier, but not enough to matter in the long run, and it encourages bad habits. If you don't need the O(1) access thing for backwards compatibility reasons, don't bother.

I did not know that Unicode itself was variable length. That alone nullifies motivation to use codepoint arrays. Why would they do that??

To limit combinatoric explosion. Without combining codepoints, you need a version of each base symbol for each combination of modifier. And while latin scripts usually limit themselves to one diacritic per base[0], other scripts (hebrew or arabic come to mind) can pile up multiple diacritics.

[0] not that that's always the case, the navajo alphabet can have both ogonek and acute on the same latin base character

I see... speaks to my ignorance on language.

Have you ever written your own unicode library, or are you just parroting what you've heard?

I have written my own unicode library, normalizing strings as UTF-32 is MUCH MUCH MUCH easier than trying to do the same in UTF-8 or UTF-16.

not to mention casefolding, and it also allows you to design better interfaces, so that the actual algorithm only needs to be implemented once.

Sounds like your library is the poster-child for:

> it blows up memory usage for ASCII and BMP (4 bytes per codepoint versus 1~3)

That would require an array of 32 bit integers (basically UTF-32). It wastes a lot of space for most characters and doesn't help that much with indexing since user perceived characters can still be composed of many codepoints. Different languages take different approaches.

Python 3.3+, I believe, looks at the string and chooses ASCII/UCS-2/UCS-4 based on the character that takes the highest amount of bytes [1]. Elixir uses UTF-8 for strings (but you can get the codepoints easily), whereas Erlang uses the list of unicode ints style.

[1] https://www.b-list.org/weblog/2017/sep/05/how-python-does-un...

Thanks for that link. It is very informative.

It's always tradeoffs.

If by "arrays of codepoints" you mean "each element of the array is a single codepoint", then you get O(1) indexing regarding codepoints, but may use up to 4x the memory as the variable-length encoding.

Which is fine when you're doing nothing but a simple operation on a string, then reencoding it.

Seriously, how many strings have you ever ran across that are more than 1kb?

how many strings have you ran across that are more than 256 codepoints?

People routinely read text files larger than 1 KB into memory. HTTP responses are often larger than 1 KB. Serialized data (JSON, XML, etc.) is often larger than 1 KB.

HTTP headers are not anywhere near 1kb of text.

the packet's themselves are only 1.5kb.

The headers are one component of a response. The body is often larger than 1 KB.

I’m currently working with a 2 MB string, representing 150k words, with a structure that is a Packed Trie stored as an array for fast access of Unicode Scalars in Swift

Sure. It's fine sometimes, and not fine other times. That's why it's called a tradeoff.

I see strings larger than that all the time, and care about using low RSS. YMMV.

Yes, it gives you flexibility in using UTF-8 and UTF-16 (which despite being a shitty format, you WILL have to support UTF-16), and also overlong sequences don't matter.

> which despite being a shitty format, you WILL have to support UTF-16

Luckily not. If you absolutely must deal with legacy encodings, feed them through a conversion pass beforehand.

Generally no. You will know when you need it.