Re:Show HN: High-speed UTF-8 validation in Rust
by duckerude on Wednesday April 21, 2021 @11:13AM
(#26890723)
> preallocating an exactly correct size result buffer
Looks like it just uses the size of the original slice. If the average broken chunk is less than three bytes (maybe quite common?) then it'll have to grow the buffer, at least doubling it.
>> let bytestring = b"foobar\xcc";
>> bytestring.len()
7
>> let cleaned = String::from_utf8_lossy(bytestring).into_owned();
>> cleaned.len()
9
>> cleaned.capacity()
14
Looks like it just uses the size of the original slice. If the average broken chunk is less than three bytes (maybe quite common?) then it'll have to grow the buffer, at least doubling it.