Speculation, but I suspect that Microsoft explicitly put both user data and programs in paths with long filenames and spaces to help force software to support those things. That always seemed like a good thing to me, it's ridiculous that we still have software that breaks on paths with spaces.