But wouldn't they be able to infer what you clicked by the data loaded?

If you block that data, most webapps, especially things like an app store, will be useless.