News for hackers. Stuff that matters.
Re:Claude can now search the web by mediumsmart on Thursday March 20, 2025 @01:23PM (#43426116)
I know an artist that had noindex turned on by mistake in robots.txt for the last 5 years - google, kagi and duckduckgo find tons of links relevant to the artist and the artwork but not a single one from the website.

so not seem to or apparently but matter of fact like. robots.txt works for the intended audience

Re:Claude can now search the web by Aloisius on Thursday March 20, 2025 @02:22PM (#43426975)
Not being indexed is different from not being crawled.
Re:Claude can now search the web by ysavir on Thursday March 20, 2025 @01:53PM (#43426551)
AI crawlers are part of the intended audience.