>I don't even understand the theory of how there is a >profit motive here. > >How does Google make more money by asking webmasters to >hide parts of their site from the indexer?

Maybe they get paid by competition. After all removing competition with clever TOS tricks is right up Googles alley:

http://pando.com/2014/05/29/after-google-bought-nest-it-remo...

Looks like google saw what bookface was doing and copied. Remove from existence and charge for exposure.