It's about what companies can afford and are willing to purchase as well.

I doubt the technology to have higher density and more energy efficient battery is far off, but I fail to see any report of mass manufacturing of these, only article about new technology advancement in labs. And if there is no incentive from buyers (the device manufacturer) for better parts (in this case the battery), then the alternative is just have something that is good enough and that won't be dwarfed by the competition.