Mandarin is based on the Northern topolect 北方话 but not identical with it. It’s a koiné[1] a dialect that emerged from communication between people who spoke mutually intelligible dialects, like Shamghainese for its river delta. That’s one reason it’s different. The other is the substantial influence from Nanjing Mandarin which would have been considered higher prestige than Beijing Mandarin into the 1800s. Beijing is the biggest influence but Mandarin is really the lingua franca of educated Inperial officials, many of whom would have come from areas where Mandarin doesn’t have retroflex r and of people for whom any variety of Mandarin was a foreign language, learned in adulthood. Lots of these people would just think the 儿话 is for peasants.

[1]https://en.wikipedia.org/wiki/Koiné_language