The point you're missing here is that as there isn't a legal standard for this, robots.txt has arguably become a de-facto standard. It's widely understood that the things you mark as off-limits to crawlers in robots.txt should not be indexed or reproduced in automated fashion.