I really doubt that Google were constrained for technical reasons. XMPP is completely extensible. I've worked with it for over a decade and there's absolutely no technical reason why Google couldn't have extended the XMPP specification and put out their own XEPs (XMPP Extension Protocol) which would have given them all the functionality and features they needed.

XMPP's only major downside at the time was mobile battery life due to XMPPs limitations over cellular. This could easily have been resolved by using a non-XMPP accepted handshake (as WhatsApp did) or by using a completely different protocol for mobile as Facebook did for mobile (MQTT).

Google decided to go for a walled garden approach as a competitive advantage. There's no technical reason for it.