News for hackers. Stuff that matters.
Re:Example: Why your Title to URL algorithm shouldn't chop o... by adulau on Monday June 07, 2010 @02:50PM (#1411381)
URL normalization/canonicalization is already a hard work but chopping off URL to avoid dirty words in any language looks also difficult. How could we implement that?

First, we need to know the existing dirty or vulgar words in a specific language (and maybe a region). Do we have WordNet for that? in any language? with a classification, a dirty word is not the same as an insult but sometime an insult can be a dirty word. I don't know any good reference of that. There are some websites with insults but a good dictionary...

Now if we look at the algorithm to do so, you will also need to know the language used in a page. It's often fine to have a French speaking page including a chopped off URL containing "cum" but not for English.

Wait a minute? Is this a problem really important? At the end, it's the only way to publish a recipe on HN...

Re:Example: Why your Title to URL algorithm shouldn't chop o... by harpastum on Monday June 07, 2010 @03:06PM (#1411446)
A simple enough solution would be to simply not include partial words. They don't help SEO, and when your phrase is long enough to require it, it's unlikely you'll have uniqueness issues.