Re:Stop Scraping My Git Forge
by lambdaxyzw on Monday June 03, 2024 @02:52PM
(#40566086)
Is there a reason why you don't? Is it just general bitterness and cynicism? As far as I know all major search engines respect rebots.txt, I don't see why LLM scrappers would be different.
[ Parent
| on HN ]
probably, yes. But coming from LLM scrappers, I have absolutely no faith in any of them. When one of them calls themself "open" in their name and is anything but, why would I trust them for anything after they lie in their name?
I also do not trust Google only crawls what is allowed in robots.txt. Maybe they only use the data allowed in public use, but I have no faith that they don't have crawled data in their version of shadow profiles.
I do not trust bigTech at all, and for those that do, I really don't understand why you do.