News for hackers. Stuff that matters.

A comment on Validating UTF-8 strings using as little as 0.7 cycles per byte. Parent comment

Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by steveklabnik on Wednesday May 16, 2018 @01:04PM (#17084113)
It's always tradeoffs.

If by "arrays of codepoints" you mean "each element of the array is a single codepoint", then you get O(1) indexing regarding codepoints, but may use up to 4x the memory as the variable-length encoding.

[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by bumblebritches5 on Wednesday May 16, 2018 @02:49PM (#17085286)
Which is fine when you're doing nothing but a simple operation on a string, then reencoding it.

Seriously, how many strings have you ever ran across that are more than 1kb?

how many strings have you ran across that are more than 256 codepoints?

[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by nirvdrum on Wednesday May 16, 2018 @03:30PM (#17085663)
People routinely read text files larger than 1 KB into memory. HTTP responses are often larger than 1 KB. Serialized data (JSON, XML, etc.) is often larger than 1 KB.
[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by bumblebritches5 on Wednesday May 16, 2018 @03:53PM (#17085871)
HTTP headers are not anywhere near 1kb of text.

the packet's themselves are only 1.5kb.

[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by nirvdrum on Wednesday May 16, 2018 @04:21PM (#17086104)
The headers are one component of a response. The body is often larger than 1 KB.
[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by mromanuk on Wednesday May 16, 2018 @10:13PM (#17088167)
I’m currently working with a 2 MB string, representing 150k words, with a structure that is a Packed Trie stored as an array for fast access of Unicode Scalars in Swift
[ Parent | on HN ]
Re:Validating UTF-8 strings using as little as 0.7 cycles pe... by steveklabnik on Wednesday May 16, 2018 @03:26PM (#17085622)
Sure. It's fine sometimes, and not fine other times. That's why it's called a tradeoff.

I see strings larger than that all the time, and care about using low RSS. YMMV.

[ Parent | on HN ]