Wouldn't it be a bad thing, even if it didn't require any privacy invasion?

If it matched human intellectual productivity capacity, that ensures that human intelligence will no longer get you more money than it takes to run some GPUs, so it would presumably become optional.