Re:Claude can now search the web
by otikik on Thursday March 20, 2025 @01:17PM
(#43426014)
Apparently, the regular search crawler does it, but the ai thingie doesn't.
[ Parent
| on HN ]
Can confirm. My website is flooded with AI bots despite attempts to block crawlers to certain parts of it.
Re:Claude can now search the web
by supriyo-biswas on Thursday March 20, 2025 @01:19PM
(#43426047)
Huh? You can add Google-Extended[1] to opt out from Generative AI summaries.
[1] https://blog.google/technology/ai/an-update-on-web-publisher...
Re:Claude can now search the web
by micromacrofoot on Thursday March 20, 2025 @01:41PM
(#43426343)
Google will still scrape it for training data either way, this only impacts search results.
Re:Claude can now search the web
by supriyo-biswas on Thursday March 20, 2025 @02:17PM
(#43426908)
> Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help *improve Bard and Vertex AI generative APIs*, including future generations of models that power those products.
Re:Claude can now search the web
by micromacrofoot on Thursday March 20, 2025 @03:57PM
(#43428126)
https://www.theverge.com/news/630079/openai-google-copyright...
they're literally asking to break laws to train AI for national security. A sentence in a press release from 2 years ago is worthless... look at what they're actually doing