'Law' is not just written legislation handed down from legislatures and other 'sovereigns'. It's also made on the ground, by custom and consensus, and then recognized as wise and just by the courts in the case of disputes – at least in common-law jurisdictions like most of the English-speaking world:

http://en.wikipedia.org/wiki/Common_law

Robots.txt has a lot of custom and implied authority behind it by now – and as others have noted, some recognition by courts as well.