That's a hard problem, and avoiding it in every situation would require scanning the strings for surrogates beforehand, when you might never need to know that information. Go makes it explicit that knowing the exact character position and string length in characters comes at a cost.
There's a good discussion of this on Tim Bray's blog: http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF
You can even export them without Capital letters ;-)
// Check that de ā” 1 mod p-1, for each prime.
// This implies that e is coprime to each p-1 as e has a multiplicative
// inverse. Therefore e is coprime to lcm(p-1,q-1,r-1,...) =
// exponent(ā¤/nā¤). It also implies that a^de ā” a mod p as a^(p-1) ā” 1
// mod p. Thus a^de ā” a mod n for all a coprime to n, as required.
Sadly, the spec requires identifiers to be just Unicode letters and digits, so we will never experience the power and glory of emoji function names in Go.Note that many languages that appear to offer indexing by rune (e.g., Java) do not in fact do so, since their 16-bit "character" type is incapable of representing all runes. The fact that this is only rarely an issue points at the fundamental rarity with with code needs to deal with runes-qua-runes.
Perhaps I should have left out the word "internal" since it's exposed.
Microsoft had reasons to pick UTF-16 for C# and the CLR. The Windows API speaks UTF-16, and when C# was first announced way back in 2000, UTF-8 was not yet a widely used encoding on the Web; Unicode was sadly not that widely used on the Web, period. The decision to use 16-bit Unicode in the Win32 API went even further back, to the development leading up to NT 3.1's release in July 1993. At that point, UTF-8 was a relative baby; it was presented at USENIX in January 1993. Also, back then, code points basically were 16 bits because surrogate pairs were but a twinkle in the Unicode Consortium's eye. The Unicode 2.0 standard added surrogate pairs to help them expand their CJK selection and generally let them add more chars of all sorts; it wasn't released until 1996. Now here we are and we have š, U+1F603.
Go came along in 2009; by then, many Web sites were being served in UTF-8, and Go'd be used in significant part in Web operations, and UTF-8 was also the default encoding in many Unix environments. Go initially didn't run on Windows at all and was ported by the community, so fitting in with Win32 wasn't an issue. Code points that wouldn't fit in 16 bits were a fact of life by '09, too. Arguments about inherent merits of encodings aside, it probably would have seemed to lots of folks that UTF-8 was a natural choice for that task at that time.
Also, Pike and Thompson are two of the three co-designers of Go and co-designed UTF-8, and UTF-8 was the encoding used by the Plan 9 OS/environment they built at Bell Labs, so, again, technical details aside, it was kind of a foregone conclusion which encoding they'd build the language around. :)
The one thing I do not want to do here is get in an argument about the inherent merits of character encodings, so I'm just not gonna do that. :)
On UTF-16 and Windows NT: http://support.microsoft.com/kb/99884, http://en.wikipedia.org/wiki/Windows_NT_3.1, and http://en.wikipedia.org/wiki/Unicode
On Unicode adoption on the Web, UTF-8, and Plan 9: http://googleblog.blogspot.com/2012/02/unicode-over-60-perce..., http://en.wikipedia.org/wiki/UTF-8, http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs
In C# you can encode/decode strings to byte arrays based on your desired encoding, but a string is composed of characters, it's in memory representation is abstracted.
Is this a performance or zero copy thing? Not having to encode/decode to get to the bytes?