News for hackers. Stuff that matters.
Re:Spell check your entire website in one go by lkozma on Monday March 23, 2009 @04:14AM (#528226)
Crawling should respect the excluded pages in robots.txt and rel="nofollow" directives.
Re:Spell check your entire website in one go by zemaj on Monday March 23, 2009 @05:45AM (#528286)
That's a tough one. This has been much debated, but at the end of the day we decided to go with our own filtering system since robots.txt and rel="nofollow" are created with search engines in mind, not spell checking. In general you want search engines to cover a subset of what you want spell checked.

You can filter on any CSS3 selector & a[rel="nofollow"] would block no-follow links if you don't want them included. By default we filter out any block with a class or an id that contains "comment" which does a pretty good job of filtering a lot of user comments.

Re:Spell check your entire website in one go by dangoldin on Monday March 23, 2009 @08:53AM (#528450)
You could always just have an option for it.

I feel people would trust your tool more if they are able to tell it what to ignore.

Re:Spell check your entire website in one go by zemaj on Monday March 23, 2009 @08:05PM (#529438)
That was the intention of choosing CSS selectors over robots.txt or nofollow, although a checkbox to specify following robots.txt & nofollow would be useful as you say.
Re:Spell check your entire website in one go by fizx on Monday March 23, 2009 @03:17PM (#529041)
Which lib are you using for selector filtering?