I think a large part of it was pin count optimization; Intel had weird beliefs about keeping the pin count down. (E.g. Intel was really resistant about using even 18 pins for the 8008, which was way too few.) Another issues was the 8086 supported two different bus protocols: "minimum" and "maximum" mode, where "minimum" was straightforward and easy to use, while "maximum" provided much more information but needed another chip to decode the signals. (I don't know if anyone used "minimum" mode.) Finally, the 8087 needed a lot of information about what was going on inside the 8086's prefetch queue, so that needed more bus signals.
They might seem weird by today's standards, but by late 1970's standards, a '40 pin' package was significantly cheaper for others to incorporate into their designs than another size. There were a lot fewer "standard socket sizes" available back then, and anything that wasn't already available entailed a lot of expense to manufacture, resulting in a very high initial cost (comparatively) to recover the capital expenditure to bring the new size to production.
http://archive.computerhistory.org/resources/text/Oral_Histo...
Well, look at that. It’s this ad: https://www.flickr.com/photos/mwichary/3236356042 and I was 11 at the time.