News for hackers. Stuff that matters.
Re:Show HN: Open-source search engine with 2bn-page index by pjc50 on Tuesday September 13, 2016 @08:22AM (#12487460)
> seems possible for a computer to detect content which is just mostly marketing

But based on the spam race, marketers will then tune content so that it doesn't trip those filters. Paid news and journal articles, etc.

Re:Show HN: Open-source search engine with 2bn-page index by GFischer on Tuesday September 13, 2016 @05:48PM (#12492538)
Extremely relevant XKCD:

https://xkcd.com/810/

But yes, even here on HN there are problems distinguishing between legitimate articles and paid news.

Re:Show HN: Open-source search engine with 2bn-page index by thr0waway1239 on Tuesday September 13, 2016 @08:52AM (#12487702)
I agree. Its a pretty tough problem, however it is good to cross the bridge when it comes. If the search engine stays really niche, perhaps it may not even be worth it for the spammers, while doing enough to cater to the somewhat self-selecting audience. For example, the number of people who want to get to the front page of HN is likely to be a really minuscule fraction of people wanting to get to the top of search results.

Also, I wonder if is it possible to detect promotional content by analyzing things like call to actions and such?

Re:Show HN: Open-source search engine with 2bn-page index by hyperpape on Tuesday September 13, 2016 @09:32AM (#12488017)
Interesting that if this were a parsing problem (i.e. https://news.ycombinator.com/item?id=12478538), folks would immediately suggest accepting known good output, instead of trying to blacklist specific problems. The analogue in search would be something that looks more like a directory than an internet wide search engine.

Of course what killed directories in the early web is that they had no hope of scaling.

Re:Show HN: Open-source search engine with 2bn-page index by pbhjpbhj on Tuesday September 13, 2016 @09:35AM (#12488037)
If they hide their sell links, shipping baskets, closing pages and such then they'll kill the sales though.

You can have paid news, but it's not doing anything if the mark can't buy the product afterwards because you had to remove all associations with selling to get the news to rank.