Agreed. Am currently in the SE 2020 and its battery life is terrible. Apple is expected to announce a SE 2022 and I'm curious how the battery life will be with it supporting 5G and a better SoC. If it's the same/worse, I'm going to upgrade to the 13 mini.
I briefly had a 6S, before I grew so sick of the size that I got my current SE. The SE’s battery is 90% the size of the 6S battery, and the 7/8/SE 2020 only increases the battery size incrementally past that point. The SE 2016 is about a mm thicker than those phones and that makes all the difference.
The Sony Xperia Compacts routinely had great battery life compared to the larger models, too — all for an extra mm or two of thickness that I, at least, cannot perceive. Phone manufacturers are just so hooked on making “7mm” phones with a 2mm camera bump they’ve forgotten that they could just omit the camera bump, curve the edges a bit if necessary, and dramatically improve battery life.
having a smaller display (and possibly downgrading/downclocking the gpu accordingly) saves some power, but the power draw of other components doesn't scale with the size of the screen. the display doesn't account for enough of the overall power budget to offset having less energy to power everything else. it's possible to make a small phone with "acceptable" battery life, but they'll never compete with bigger phones on that front.
btw, apple doesn't have some magical solution to this scaling problem. their small phones have significantly worse battery life than their larger offerings. it's just that their overall efficiency advantage makes a small iphone more viable than a small android phone.