I'd say a JavaScript programmer should think of things this way:
* JavaScript strings are exposed to the programmer as an array of UTF-16 code units, although there are some helper functions like `codePointAt` to help interpret strings in terms of code points.
* Newer languages expose strings as an array of Unicode code points, which is cleaner because it's independent of any particular encoding.
* Even when working with code points, you can't safely reverse strings or anything like that, since a user-perceived character might consist of multiple code points.
It is still very important in general to distinguish a string from a byte array, and I felt like the article was fairly counter-productive with all its talk about "UTF-8 strings" (which doesn't make much sense, either you have a byte-array that you can apply UTF-8 decoding on to get a string out of, or you already have a string, in which case encodings in the traditional sense (UTF-8, ISO-8859-1 latin1, etc) doesn't apply)
Eh. I agree and disagree. I agree in the sense that the phrase "UTF-8 string" is generally a misnomer and is a good signal that there's some confusion somewhere, but I don't find think I find it as damning as you do. In particular, not all languages represent strings as sequences of codepoints, and instead make their internal UTF-8 byte representation a first class part of their string API. Two languages that come to mind are Go and Rust, where Go's conventionally uses UTF-8 and Rust uses enforced UTF-8. But in both cases, accessing the raw bytes is not only a standard operation, but is necessary whenever you want to do high performance string processing.
That is, if someone said Go/Rust had "UTF-8 strings," that wouldn't be altogether wrong. UTF-8 is a first class aspect of both string APIs, while both provide standard functions you'd expect from Unicode strings.
But internally, JavaScript engines can use whatever encoding they like, though most use UCS-2 most of the time. (Servo, for example, represents documents in WTF-8 rather than the conventional UCS-2. This is good for memory efficiency and certain other things, though bad for arbitrary indexing which is rare.)
Python 2 is in the same boat concerning internal representation (it will use UTF-16 much of the time), but it doesn’t expose any UTF-16ness in the API.
Don't do that (the array part). Guaranteeing O(1) indexing isn't very useful but precludes efficient storage.
You imply that there is some universal "string" definition, when there is no such thing.
Literally, "string" just means sequence. Beyond that, it depends on the context.
* C++ std::string is a sequence of char (aka byte, defined to be 8+ bits).
* Swift string is a sequence of Unicode graphemes.
* Ruby String is a sequence of bytes/octects, with an attached (and mutable) encoding.
* Python 2 str is a sequence of bytes/octets
* Python 3 str is a sequence of Unicode code points
* ECMAScript string (and Java java.lang.String) is a sequence of UTF-16 code units.
Moreover, the entire idea of "Strings are an array of integer unicode code points" is confined to Unicode. It doesn't say anything about other character sets, e.g. ASCII, ISO 8859-1, or Windows-1252. (Though AFAIK Unicode is a superset of those particular three.)
So...I think you can safely say "Unicode strings are sequences of Unicode code points."
I'd argue that, generally, strings are simply arrays of chars, which are bytes.
THe failure here, was keeping the name "string" for what are arrays of codepoints instead of bytes.
Unicode strings are arrays of code points which are 21bit numbers.
If the API requires fast subscript (it usually does) then they would be UTF-32 or three-codepoints-in-int64, otherwise more compact internal representation is possible.
If you don't require supporting subscript and allow only iteration over list of code points then in-memory representation of strings can be more compact. It can use UTF-8 or even SCSU or BOCU1.
Some languages use polymorphic unicode strings which store ascii if the value is all-ascii and switch to something else if it isn't (python3.3 and factor come to mind).
In C, strings were always semantically a sequence of characters (as they are commonly defined elsewhere).
For a while a character was one byte, so the distinction was unimportant (and became blurred).
A char was both a character and a byte. A string was both a sequence of characters and an array of characters... and an array of "char"s, and an array of bytes.
Code -- and programmers! -- became dependent on these equivalency assumptions.
Once it became clear we could no longer pretend that a maximum of 256 characters was tenable (less actually, since the use of 0-31 for control/separation/termination had become standard) we were left with conflict, leading to a variety of uncomfortable choices and compromises.
One such conflict is "char"... should it retain its semantics or its size (one byte)?
The last time I developed seriously in C or C++ it had retained its size, but lost its semantics -- a char IS a byte now. (That was a while ago, I don't know if that's changed -- it sounds like from your post that it hasn't.)
I guess UTF-8 has won out in C and C++ (and elsewhere) so now, while a char is byte, a C/C++ string is: (1) an array of char/bytes; (2) a sequence of characters. The thing that's been dropped is that a string is no longer an array of characters.
(In case there's confusion: here, "array" means an ordered sequence of elements of uniform size with O(1) random access, while sequence is just an ordered sequence of elements that doesn't necessarily offer O(1) random access or elements with uniform size.)
So, for example, I can figure out how to take a document written in Microsoft Word with that Latin-1 business and make the characters stop sucking in python 3, but I don't even know what to google to do the same thing in javascript, because people use terms like "encoding" and such totally differently.
No such thing! Strings are an array of integer unicode code points. Stop thinking about bytes at this level. The internal representation of strings and chars does not matter, because you as a programmer ever only see integer code points.
Encoding only enters the picture the moment you want to convert your string to or from a byte array (for example to write to disk or send over the network). The encoding, such as "UTF-8", then specifies how to map between the array of abstract code points to an array of 8bit bytes.