[1] At least when talking about code points, which is what matters for regular expressions (unless you want stuff like \X with is not universally supported).
The standard '.' (match any character) in a regexp matches an Unicode code point, not a grapheme cluster. To match a grapheme cluster, you have to use "\X", which is not universally supported.
For example, in Python 3, the builtin module module "re" doesn't support "\X", you have to install the "regex" module for it:
# text is 'e' followed by U+0301 (combining acute accent):
text = b'e\xcc\x81'.decode('utf8')
print(f'text: "{text}"')
import re
print(re.match('^(.)(.)$', text).groups()) # prints "('e', '´')"
import regex # must be installed
print(regex.match('^(\X)$', text).groups()) # prints "('é',)"You have the words "supported" and "implemented" mixed up. Kernighan claims Unicode support, so he is required by the standard the implement \X.
If a software does not implement \X, then it is not compliant, and it would be very wrong to say it supports Unicode. Does anyone have a deeplink showing the evidence for awk?
For example, I regularly say that Rust's regex crate has Unicode support. But it does not support \X. It's more precisely documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md
Which standard is that?
If you're talking about Unicode, the "standard" for regular expressions[1] is an "Unicode Technical Standard", which according to itself isn't required for Unicode conformance:
> A Unicode Technical Standard (UTS) is an independent specification. Conformance to the Unicode Standard does not imply conformance to any UTS.
So awk can claim Unicode support without supporting "\X" (like many regex engines).
If you're talking about POSIX, its regex chapter[2] doesn't mention "\X". In any case I don't think awk claims to conform to POSIX.
[1] https://unicode.org/reports/tr18/
[2] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
I think I once created one that was about a kilobyte. Is there an upper limit?
I created it using this page https://glitchtextgenerator.com/
The 73 byte X:
x̧̡̬̘͓̖̲̻̻̲̠̪̻͓͙̜̂̓̊̔̀̀͗̑̀̅̀̂̚͘̕̚͘͢͜͠
However you'd probably like to store something more compact than, say, JSON arrays of integers. So there are also a bunch of encodings which turn the integers into bytes. These encodings would work for any integers, but they make most sense to encode Unicode's code points. UTF-8 turns each code point into 1-4 bytes, a pair of UTF-16 encodings turns them into one or two "code units" each of 2 bytes either little or big endian. And UTF-32 just encodes them as native 32-bit integers but again either little or big endian.
> A: Unicode is the universal character encoding, maintained by the Unicode Consortium. This encoding standard provides the basis for processing, storage and interchange of text data in any language in all modern software and information technology protocols.
1. Map a character to a unique number in a character set (in Unicode: called codepoint)
2. Map a number that represents a character in a character set to a bit pattern for storage (transiently or persistently, internally or externally). Unicode code points can be bit-encoded in various ways: UTF8, UCS2 and UCS4/UTF32.
The original code points permit the same character to be represented in various ways, which makes equality checks non-trivial: for instance a character like "ä" can be represented as a single character or alternatively as a composition of "a" + umlaut accent (2 characters).
So far, this is all about plain text, so we are not talking about font families or character properties (bold, italics, underlined) or orientation (super-script, sup-script).
Ken Lunde's opus magnum is the standard book on representing text in various languages other than English, with a focus on Asian languages: https://www.oreilly.com/library/view/cjkv-information-proces...
It’s like a catalogue of scripts. We have to extend it when we encounter new scripts that are not catalogued yet (or when we create new emojis)
Converting a byte sequence to a Unicode code point sequence and vice-versa is called transformation format (or more generally an encoding form, but then might not be deterministic) by Unicode (see <https://www.unicode.org/faq/utf_bom.html#gen2>). Unicode specifies UTF-8, -16 and -32. We do not have to change these formats unless the catalogue hit the limits of 32 bits (not a big problem for UTF-8 but for the other two formats). These formats are already able to encode code points that are not assigned yet.
And the confusion now is that a lot of people call what Unicode calls transformation format (i.e. the byte to code point mapping) encoding as well. The term charset is also used sometimes.
PS: Note that a goal of Unicode is to be able to accommodate legacy encoding/charsets by having a broad enough catalogue. This is so that these legacy encoding which may come with their own catalogue can be mapped to the Unicode catalogue. So we have control codes (even though not part of any “proper” human script), precomposed letters (there is a code point for à although it could be represented by a + combining `), things like the Greek terminal form of sigma separately encoded, although that could be done in font-rendering (like generally done for Arabic), and a lot more to aid with mapping and roundtrips.
Edit: by which I mean that I’ve only seen the non terminal form so far in maths but it’s hard to write an algorithm that distinguishes between a word ending in sigma and some juxtaposition of Greek variables.
That's not actually useful, because unicode itself is a variable length encoding.
So it mostly blows up the size of your data.
Though it might have been selected for implementation simplicity and / or backwards compatibility (e.g. same reason why Python did it, then had to invent "flexible string representation" because strings had become way too big to be acceptable).