> So it seems like Apple was looking to build their own processors even before the iPhone became huge (the original iPhone only sold around 7M units while iPods were selling around 50M/year at the same time).

This is almost certainly true once they decided that iOS was going to be based on MacOS. Those software design choices had to have been based on a very aggressive hardware roadmap