That's a sloppy mistake and should without doubt have been caught in testing.
"It was definitely a mistake on our part. The problem was not the encoding on our data feed, but the HTML document was sent with ISO-8859-1. The second we inserted the twitter text into the DOM, the browsers interpreted the UTF-8 string as ISO-8859-1. Our visualizations are hosted on other platforms, and in this case the server was not configured to send UTF-8 with text/html even though the HTML file was encoded as such. It was the only issue (albeit a pretty obvious one) during an otherwise flawless event. I apologize to President Obama, Speaker Boehner, and Jack Dorsey for the mistake. If the readers of the blog think it was stupid, imagine how we felt. dev environment != production environment. If we would have just included a <meta charset="utf-8"> in the HTML head, then this would not have occurred.
The big take away is don’t make assumptions about other platforms (especially when it comes to encoding), and always include charset meta tag." [emphasis mine]
Something like this could be done as nginx/apache module: which detects encodings of the data and transcodes the HTML output into utf8 - could be useful for some cases.
Every time you don't validate...God kills a kitten. Please, think of the kittens.
This has been a Public Service Announcement. Please code responsibly: http://validator.w3.org/
<meta charset="utf-8">Edit: yes, there were definitely people curating the questions, so it could not have been as simple as pulling one JSON feed. http://www.theatlantic.com/politics/archive/2011/07/how-obam... https://twitter.com/#!/townhall/july-6-curators
...That's kind of the point. There was someone being sloppy there.
There is no excuse for not testing your app for basic i18n brokenness. It's 2011, not 1998.
Oh come off of it. This happens everywhere on the web on probably something like 25% of websites. And it's NOT always the consuming program's fault: very often somebody upstream, e.g. the hosting company, the person that wrote the HTML, the source of an RSS feed being inserted into the page etc. etc. forgot to encode something the way somebody else expected, and you as the poor guy at the end of the chain gets a document with multiple encodings improperly embedded into it. Inevitably you have to make some bad decisions and not all corner cases are handled.
Somebody once reverse-engineered the state chart for how Internet Explorer handles documents with conflicting encoding declarations and I kid you not, it must have had >20 branches spanning a good few pages. Officially, the correct order of precedence is (http://www.w3.org/International/questions/qa-html-encoding-d...):
1. HTTP Content-Type header
2. byte-order mark (BOM)
3. XML declaration
4. meta element
5. link charset attribute
but that's not how every browser does it, because the W3C sort of declared that after things on the Real Internet (TM) had already gotten out of hand. I hate to resuscitate Joel posts but Unicode is not easy to implement right.