Re:Claude can now search the web
by mediumsmart on Thursday March 20, 2025 @01:23PM
(#43426116)
I know an artist that had noindex turned on by mistake in robots.txt for the last 5 years - google, kagi and duckduckgo find tons of links relevant to the artist and the artwork but not a single one from the website.
so not seem to or apparently but matter of fact like. robots.txt works for the intended audience
so not seem to or apparently but matter of fact like. robots.txt works for the intended audience