News for hackers. Stuff that matters.
Re:LLaMA2 isn't “Open Source” – and why it doesn't matter by makeitdouble on Friday July 21, 2023 @04:01PM (#36818899)
Google respects the "robot.txt" and asks you to use it to opt out of their crawling.

Parent's point is if your own scaping army respects the "scaping.txt" and goes down on Google as they don't opt-out in their scraping.txt, it probably wouldn't fly.

Re:LLaMA2 isn't “Open Source” – and why it doesn't matter by kgwgk on Friday July 21, 2023 @04:47PM (#36819494)
I don't understand. What does "Rules for thee but not for me" mean if "google is allowed to scrape" whatever people allows Google to scrape but "you’re not allowed to scrape google" because using the same rules google.com/robots.txt says

   User-agent: *
   Disallow: /search
   ....
Re:LLaMA2 isn't “Open Source” – and why it doesn't matter by makeitdouble on Friday July 21, 2023 @05:12PM (#36819852)
There's an imbalance because the robot.txt rule is something Google pushed forward (didn't invent it, but made it standard) and is opt-out. So yes, Google made up their rules and won't let other people to make up their own self-beneficial rules in a similar way.
Re:LLaMA2 isn't “Open Source” – and why it doesn't matter by kgwgk on Friday July 21, 2023 @05:24PM (#36819984)
> Google [...] won't let other people to make up their own self-beneficial rules in a similar way.

What "other people"?

If it's the "you" who is not allowed to scrape google in https://news.ycombinator.com/item?id=36817237 then you can make your own "google is not allowed to scrape my thing" rules if you think that's beneficial for you.

If it's somehow related to LLM providers or users I doubt that's what the original comment was referring to.

To be clear, I understand the original comment as

   LLM companies say "I can use what you generate and you cannot not prevent me from doing so, but I won't allow you to use the output of the LLM" just like Google says "I can scrape what you generate and you cannot not prevent me from doing so, but I won't allow you to scrape the output of the search engine"
and that doesn't seem a valid analogy.
Re:LLaMA2 isn't “Open Source” – and why it doesn't matter by makeitdouble on Friday July 21, 2023 @10:14PM (#36822507)
You should change "you cannot prevent me from doing so" into "you'll need to setup your ressources in the way that I defined if you don't want me to slurp them".

I see it as the equivalent of the spam mail that require the user to login to disable them.