knight666
Windows 10 Broke My Unit Test
Utf8rewind 1.4.0 – Adds support for case folding and matching by category
Utf8rewind's case mapping is 75% faster and now supports Turkish
Utf8rewind 1.3.0 adds Turkish case mapping and improves performance by 75%
Utf8rewind 1.3.0 – 75% faster and now supports Turkish, Lithuanian and Greek
The Curious Case of the Greek Final Sigma
#define UTF8_VALIDATE_PARAMETERS(_inputType, _outputType, _result) \
if (input == 0) { \
UTF8_SET_ERROR(INVALID_DATA); \
return _result; \
} \
If a NUL parameter is passed to utf8toutf16 and friends, they'll return UTF8_ERR_INVALID_DATA.However, you are correct about codepoint_read's behavior. I actually have a test for passing NUL to the function, but the value of decoded isn't checked:
TEST(CodepointRead, InvalidData)
{
const char* i = nullptr;
size_t il = 0;
unicode_t o;
EXPECT_EQ(0, codepoint_read(i, il, &o));
}
It's definitely a bug, but relatively harmless. The function is only used internally and the calling sites guard against that kind of attack.Nevertheless, I thank you for the extra set of eyeballs. I have been looking at these functions far too long, you tend to gloss over obvious problems after a while.
I'll be sure to get a fix in for the next release. :)
As you get down the list you'll notice what a pain in the ass the special cases are. There's a special case for the final sigma in a Greek word:
03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
You must remove the dot from "i" when upper or titlecasing... but only in Lithuanian: 0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
Etc. etc. By the way, my implementation for case mapping started out similarly to yours, but I ultimately solved the problem using a binary search in a huge look-up table: https://bitbucket.org/knight666/utf8rewind/src/c22e458912952...Unicode case mapping is just a huge mess of exceptions, but that's more the humans' fault than the standard.
The library currently doesn't employ any kind of fuzzing. I rely on multiple tests for every kind of input. I've found that is a pretty reliable way to test for "unknown unknowns", even if it's extremely time-consuming.
Adding testcase fuzzing is definitely something to consider though, because it would most likely have found the very issues this release fixes.
It's very thoroughly tested but it hasn't been used in any serious projects yet.