They had a simple rule that they index what users can see and if you try to cheat your way around that you get ranked down. Sticking to that for everyone would have been an iron clad defense against monopoly accusations. Instead they chose to help certain corporations (or at least corporations with certain business models) present different content to the bot and users.