Just so you know, that doesn’t stop Google from indexing you anymore. It only stops Googlebot from crawling. In other words it’s strictly a place to deny permission or set limits on a technical level. If you do this they will still have what they already crawled and if there are links to it on the Internet they will keep it in the index.

If you don’t want something to be in Google on a policy level, you now have to add the noindex meta tag on every page.