At that point, you're dependent upon Google for running your public number. If somebody comes out for a better service, or Google decides to charge money for the GV service, you might not be able to transfer your number.

I'm all for giving AT&T less control in favor of Google because Google is decidedly less evil, for now. But make sure you have protections for future events.