Re:Show HN: High-speed UTF-8 validation in Rust
by cryptonector on Wednesday April 21, 2021 @04:47PM
(#26895237)
This is SIMD based. If the encoding_rs crate isn't SIMD based, then this thing will beat its pants off.
[ Parent
| on HN ]
Re:Show HN: High-speed UTF-8 validation in Rust
by burntsushi on Wednesday April 21, 2021 @05:28PM
(#26895666)
It does use SIMD (when enabled). But also, the encoding_rs crate is doing more. It isn't just checking for validity. It's also doing transcoding and error detection.
Re:Show HN: High-speed UTF-8 validation in Rust
by kryps on Thursday April 22, 2021 @02:27AM
(#26899436)
As far as I can see it does not use SIMD for UTF8 validation, only likely/unlikely instrinsics, see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...
Re:Show HN: High-speed UTF-8 validation in Rust
by jfk13 on Friday April 23, 2021 @04:13PM
(#26918708)
That's not the only use of SIMD in the crate (e.g. see https://github.com/hsivonen/encoding_rs/blob/e98a2096ab09c92...), but I haven't looked into exactly where/how it's used further.