AFAIR the CPU/APU size is quite small. What takes quite a lot of space is battery. And because things are less and less optimize we need more computing power which needs more juice. What's more - bigger screens require more power as well.

Why those big phones popular? Because CPU efficiency doesn't sell well but number do, so we have a race: more storage, more ram, more MP camera... and bigger screen. Those are the most visible and easiest to adjust things. And many people associate size with being "better"...