Why would Microsoft agree to such a deal? Is it really so hard to take Xbox orders online and ship directly to the customer?

I suppose gamestop would then take the position of trying to sell Xbox consoles instead of PlayStation. They can even offer discounts based on their future expected earnings from the digital purchases on the console.

Gamestop has a significant retail presence, Microsoft has none. It's quite tempting to just go to the local shopping center, pick up a system and a handful of games (with a decent return policy that's more reasonable than the one at big box stores) rather than ordering it online and waiting for it to arrive.