News for hackers. Stuff that matters.
Re:The dangers of streaming across versions of glibc: A caut... by alexchamberlain on Tuesday January 08, 2019 @01:36PM (#18858087)
May I ask a stupid question? What's the difference between collation and encoding? Some people in this thread have suggested "use byte strings"; is it "enough" to "just" use UTF-8, like you should "just" use UTC (unless you know/Should know better)?
Re:The dangers of streaming across versions of glibc: A caut... by ben509 on Tuesday January 08, 2019 @04:43PM (#18860030)
To get a bit into the theory: a string is a sequence of symbols drawn from some "alphabet", and we can define operations like concatenation, counting, matching and substring on it. In particular, a natural operation on strings is lexicographic comparison, whereby you compare the first characters of two strings, if they are different, the two strings compare that way, if they match, you proceed to look at the second character, etc.

Encoding translates a string from one language to another, and it's often a reversible translation. The obvious case is to translate from Unicode to binary and back. In Unicode, the alphabet is the set of all Unicode codepoints. In binary, you have a stream of octets and the alphabet is the set of integers between 0 and 255. So UTF-8 is a standard that specifies how to perform such a translation.

Since we're often going back and forth between Unicode and binary, and because real processors work with bytes, it's reasonable to call "encoding" translating a string to binary and "decoding" translating binary to a string. The math doesn't require that, though.

Collation represents a string as an expression (usually also a string) that is trivially comparable. For instance, supposed I wanted an English-friendly way to compare band names. I might map all codepoints that look like A or E to "A", then K, S and C to C, etc., discarding any accents and maybe throwing out leading junk like "The" or "A".

That's why a case-insensitive collation won't let you have Foo and FOO, because they would both collate as FOO. It also means, unlike encoding, collation is expected to be one-way.

Re:The dangers of streaming across versions of glibc: A caut... by jancsika on Tuesday January 08, 2019 @09:38PM (#18861830)
> Collation represents a string as an expression (usually also a string) that is trivially comparable. For instance, supposed I wanted an English-friendly way to compare band names. I might map all codepoints that look like A or E to "A", then K, S and C to C, etc., discarding any accents and maybe throwing out leading junk like "The" or "A".

So how does collation algorithmic instability break an application?

Edit: What I mean is-- what are a set of reasonable assumptions a dev would make in the application layer that would be good practice on the one hand, yet also be wholly dependent on collation algorithm stability to keep from breaking?

Re:The dangers of streaming across versions of glibc: A caut... by ben509 on Tuesday January 08, 2019 @10:04PM (#18861944)
All the assumptions that make a database index work will break.

If you think about any kind of binary tree based structure, it will have invariants like "the left child node will be less than the parent node, and the right child node will be greater than the parent node."

If those are violated just a little bit, at best data will simply disappear from the index. It's possible for updates to delete stuff, or for operations on the index to hang.

Re:The dangers of streaming across versions of glibc: A caut... by BeeOnRope on Tuesday January 08, 2019 @01:39PM (#18858122)
Collation is basically a fancy way of saying "sorting" and should be independent of encoding (or course, certain encodings lend themselves more naturally to certain collation algorithms). In general collation needs to be locale-aware while encoding (in Unicode) largely doesn't care.