News for hackers. Stuff that matters.
Re:UCS vs. UTF-8 as Internal String Encoding by vardump on Thursday May 28, 2015 @04:28PM (#9620945)
> I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top.

That's right. We could figure out how to represent common characters with less bits. For low-cost character stream compression, byte granularity should work nicely with computers, so we should it. A byte could have a bit that indicates the range is extended, like highest bit, because it's easy to process. Same idea could be chained to represent full range of unicode characters. That way we'd have compressed UTF-32 characters!

Re:UCS vs. UTF-8 as Internal String Encoding by derefr on Thursday May 28, 2015 @11:34PM (#9622546)
I meant more something like RLE or simple Huffman encoding. A "string" type like Redis's dict type or ObjC's array type, where the internal representation changes as the allocated object grows in size and therefore is worth making different trade-offs on. (So, a five-byte string, maybe UTF-8 is a fine encoding. A 3KB string? Worth prepending a compression dictionary to.)