The Fall of Stack Overflow

Re:The Fall of Stack Overflow by pjc50 on Tuesday July 25, 2023 @08:03AM (#36861056)

It's an adversarial process. The search engine is, in theory, trying to rank by usefulness to the user, and the site owner is trying to maximize revenue by lying to the search engine. And the user.

[ Parent | on HN ]

Re:The Fall of Stack Overflow by XCSme on Tuesday July 25, 2023 @08:47AM (#36861437)

Then, maybe the "crawler" should be an actual PC navigating to the browser, taking a screenshot (or live feed) of the page and processing it with AI.

[ Parent | on HN ]

Re:The Fall of Stack Overflow by pjc50 on Tuesday July 25, 2023 @11:03AM (#36863352)

Eh, Google choose to be identifiable as googlebot and to obey robots.txt for other reasons of "good citizenship", because not everybody wants to be crawled.

[ Parent | on HN ]

Re:The Fall of Stack Overflow by AlexandrB on Tuesday July 25, 2023 @09:05AM (#36861605)

I'm generally puzzled by Google's reluctance to do manual intervention in these cases. It's not like this is a secret. Just penalize the whole domain for 60 days every time a prominent site lies to the crawler.

[ Parent | on HN ]

Re:The Fall of Stack Overflow by gtirloni on Tuesday July 25, 2023 @10:17AM (#36862598)

If Google's support is any indication, Google doesn't like to involve humans in their processes. There probably isn't enough humans to do this manual intervention you propose.

[ Parent | on HN ]

Re:The Fall of Stack Overflow by sulam on Tuesday July 25, 2023 @10:50AM (#36863120)

There are very many sites where the content you see as a non-logged-in user is different from what you see if you have in your possession an all-important user cookie.

[ Parent | on HN ]