We had made a point of asking before the project if regionalisation was ever going to be an issue and no, it would only ever be in English. Shortly after go live we were asked to regionalise everything into Chinese.
I'm still not sure what a capital letter looks like in Chinese.
Fun fact: there exists a convention stipulating a double space after a period that ends a sentence. Not that I'm advocating relying on this for any serious purposes
In fact I just checked, and HN is honoring the two spaces, in that they are output to the actual HTML sent to your browser. And of course, yes, there were other trends that would have ended this anyhow, "two spaces" in meaningless in a non-monospaced font and ever more stuff is going proportional as the computing power necessary to do that continues its steady march from "prohibitive" to "trivial", but the WWW certainly beat the corpse to death again.
[space]
[space]<span class="something-about-space">[space]</span>It took me forever to stop hitting the space bar twice after ending a sentence.
If you want consecutive spaces in a bare HTML page, you have to use " " which doesn't work here because pg isn't a moron.
So, one particular solution with these performance characteristics is building a decision tree using a bunch of training data, and eg. a maximum entropy classifier. Add some sample data from any openly available corpus's (or fire up mturk, and create your own), and you're pretty much done with it.
Of course, sentence-tokenization is only the tip of the iceberg :)
Some of the most common icebergs are:
-form validation (seriously -one of the most highly exercised user-interaction paths; it's all over the place, and scales semi-exponentially with the number of fields)
-search ("how hard could it be? you just put an input form there, then figure out what the user thought, then display it" -exact quote)
-anything that has to process natural language. I mean everything. Wanna split up a text into sentences? How do you differentiate between dr. mr., 2004. jun. , and valid sentence-enders? Generating a definite article ("a", "an") before a noun? Keep in mind that 1,2,@,$,=, and other characters might also be valid noun first-letters :) etc.
In my experience, the best anti-iceberg pattern is to follow a portfolio approach, and for each requirements which smells like iceberg, have a fallback plan in place -ie. after N hours of sunken investment, execution shifts to plan B. Usually works out much better, than banging away on the same problem for days.