I don't believe there's a law requiring the honoring of the robots.txt file. People and services honor the file out of a sense of good manners, not a legal requirement.

It doesn't have to be a specific law. It is a rebuttal to a claim of "I had no idea I shouldn't have requested millions of pages from that site".

If you scrape a site that prohibits it in robots.txt, that should be considered notice that they don't want that, for whatever relevant law. (I don't know if this argument would hold up in court, IANAL.)

I think I see what you're saying, but I disagree that the robots.txt file should have any legal ramifications. Web site operators have many tools that they can use to limit traffic or protect data and they should make good use of those tools.

LinkedIn wants to make their data available publicly, except under certain conditions. In my opinion, if they can't find a technical solution, they should stop making the data available publicly.

What is a robot? Why is the User Agent even important? It's not a standardized value. I could send "User-Agent: ikeboy" and it's perfectly valid.