News for hackers. Stuff that matters.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by IgorPartola on Thursday October 18, 2012 @09:27AM (#4668611)
Under what case would IE use Latin 1 when there are UTF-8 characters that should be encoded? I seem to be missing the actual effect it's having.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by Millennium on Thursday October 18, 2012 @09:46AM (#4668700)
If you specify a form's data as UTF-8, but every character the user types in happens to also occur in Latin-1, IE will disobey what you've set and send it in Latin-1 instead. In many locales -perhaps most- that's going to be a very common occurrence, since Latin-1 covers seven of Abram de Swann's twelve "supercentral languages" and three of the top ten languages by number of native speakers.

The point of the checkmark, therefore, is to put it in a hidden form field. That way, no matter what the user types, there will still be at least one non-Latin-1 character. That will force IE to use UTF-8, and you can check to make sure this actually happened by checking the value of the form field: if it's not set correctly, then you know there may be trouble.

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by pdw on Thursday October 18, 2012 @10:04AM (#4668773)
I think this only happens if the user manually switches to Latin-1 encoding. In that case IE will try to use the same encoding when submitting form data. The user might do this if you already have encoding problems and present a mixed Latin-1/UTF-8 page. The snowman hack serves to prevent the corruption from spreading.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by agwa on Thursday October 18, 2012 @11:59AM (#4669375)
> I think this only happens if the user manually switches to Latin-1 encoding.

That's correct. If you send your HTML document with a charset of UTF-8 (In the Content-Type header) then IE will submit forms using UTF-8 even if the user doesn't input any UTF-8 characters. Unless the user changes the encoding, but I have yet to hear a compelling reason why an ordinary user would do that under ordinary circumstances.

> The snowman hack serves to prevent the corruption from spreading.

It's clever, but the framework could also just reject POST and GET requests which contain invalid UTF-8 characters. (I'm flabbergasted that Ruby doesn't do this[1].) Otherwise a malicious user could try to inject non-UTF-8 characters into your database by sending crafted requests which nevertheless contain the "utf8=✓". And speaking from experience, you do not want to have to deal with encoding problems in your database.

[1] http://stackoverflow.com/questions/3222013/what-is-the-snowm...

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by Millennium on Thursday October 18, 2012 @12:22PM (#4669475)
Whether or not you use this hack, you can't naively trust the client to always send valid UTF-8; you're right about this. But because of this bug in IE, rejecting posts with invalid UTF-8 as malicious will net you some false-positive cases, where the user isn't malicious but the browser is being stupid. This hack takes care of the stupidity, leading to a better user experience for people who would otherwise have tripped the false-positive.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by Tobias42 on Thursday October 18, 2012 @05:33PM (#4671225)
What if a sequence of byte values is valid in the charset that IE uses to encode the form data as well as in UTF-8, but is interpreted as different characters in UTF-8? With your method you would not detect an error and use the wrong characters. (Except if IE sends a content-type header with the actual encoding used, and this header is evaluated on the server side to convert the form data into a string. But in that case you don't have to check for invalid UTF-8 characters, but for characters that are invalid in the charset specified in the content-type header.)
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by carllerche on Thursday October 18, 2012 @12:52PM (#4669600)
Ordinary users would do it because many corporate sites used to direct them to do this as a fix for their browsers displaying invalid characters.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by agwa on Thursday October 18, 2012 @01:00PM (#4669655)
Would they do that if their browser isn't displaying invalid characters? That's what I meant by "ordinary circumstances."
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by wycats on Thursday October 18, 2012 @03:50PM (#4670715)
Corrupted characters in a MySQL database is unfortunately quite common. The user fix will thus propagate the corruption without this fix.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by jerf on Thursday October 18, 2012 @09:36AM (#4668653)
Yes, that took me a moment too, but Latin-1 is an 8-bit ASCII, and UTF-8 only encompasses 7-bit ASCII. Note I have to say an 8-bit ASCII, because there are numerous 8-bit ASCII encodings.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by ygra on Thursday October 18, 2012 @11:09AM (#4669120)
ASCII is a 7-bit code. Latin 1 uses 8 bits and therefore cannot be ASCII. It's a superset of ASCII, though.
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by derleth on Thursday October 18, 2012 @04:20PM (#4670846)
> UTF-8 only encompasses 7-bit ASCII

What? This is wrong. UTF-8 encodes a lot more than just ASCII.

UTF-8 is compatible with ASCII in that all of the characters ASCII and Unicode have in common are represented the same way in ASCII and UTF-8. Going beyond ASCII involves the introduction of multi-byte representations in UTF-8, and that takes you smoothly (that is, no surrogate pairs) out into the entire rest of Unicode. As a bonus, it's always possible to verify that a given string of bytes is valid UTF-8, given that there is a nontrivial structure imposed on UTF-8 multi-byte encodings that is very unlikely to occur by chance in any non-UTF-8 sequence of bytes.

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by Firehed on Thursday October 18, 2012 @04:58PM (#4671083)
I think the point was that anything above 7-bit ASCII will be represented differently in Latin-1 vs UTF-8; i.e. ¢ (U+00A2) is rendered as 0xA2 in Latin1 and 0xC2A2 in UTF-8 - and 0xC2A2 in Latin1 will be displayed as ¢.

It gets far worse in 3-byte UTF8 characters, but I don't believe any of them exist natively in Latin1 (see: euro symbol)

Assuming I'm reading these various character tables right, at least ;)

So a more accurate version of what you quoted would be "UTF-8 and Latin-1 only overlap for 7-bit ASCII"

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by kbolino on Friday October 19, 2012 @08:37AM (#4673220)
Not to detract from your points, all good, but:

0xC2A2 will be rendered as ¢ only if it's encoded in UTF-16/UCS-2 big endian and misinterpreted as ISO-8859-1/Windows-1252.

If it's encoded in little endian (much more common on Intel x86 computers), then it would be rendered as ¢Â when misinterpreted.

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by kelnos on Saturday October 20, 2012 @08:27AM (#4676901)
That doesn't really make sense. If someone's intending to encode ¢ in utf-8, endianness does not come into play, as it's a stream of octets, not of anything larger that you can chunk such that you could swap bytes.

At any rate, if you were to encode ¢ in UTF-16BE, it would be 0x00a2, not 0xc2a2. If a piece of software then misinterpreted it as latin1, likely you'd get nothing at all due to the embedded NUL.

  $ echo -n ¢ | iconv -f UTF-8 -t UTF-16BE | hexdump -C
  00000000  00 a2                                             |..|
Re:Is the use of “utf8=✓” preferable to “utf8=true”? by kbolino on Thursday October 25, 2012 @08:04AM (#4697264)
"That doesn't really make sense."

Indeed. I either completely misread the parent post, or else it said something different when I responded to it (knowing myself, I'm going with the former).

Re:Is the use of “utf8=✓” preferable to “utf8=true”? by jpenner on Thursday October 18, 2012 @09:32AM (#4668633)
Some accented characters are in Latin-1, like ç or é.