News for hackers. Stuff that matters.

A comment on UCS vs. UTF-8 as Internal String Encoding. Parent comment

Re:UCS vs. UTF-8 as Internal String Encoding by vardump on Thursday May 28, 2015 @02:12PM (#9619987)
24-bit and 21-bit format wouldn't have much difference in terms of performance of constant time indexing. Strlen (for storage space) would be dramatically slower than in any other format.
[ Parent | on HN ]
Re:UCS vs. UTF-8 as Internal String Encoding by TazeTSchnitzel on Thursday May 28, 2015 @02:35PM (#9620154)
Would it really? You can still use SIMD and vectorise.
[ Parent | on HN ]
Re:UCS vs. UTF-8 as Internal String Encoding by vardump on Thursday May 28, 2015 @02:55PM (#9620303)
I don't immediately know how to use SIMD to scan for 24-bit wide zero terminator or for any character for that matter. 128/24 is 5 1/3 or 5.333... Alignment would mean a lot of slow operations to stitch data between memory loads.

How would you efficiently scan for a specific 24-bit character by using SIMD? Say you're not running on Haswell, but something older that's not as good with unaligned memory accesses?

[ Parent | on HN ]