Yes. But that's my point - with current software this works out well for me, both on the go and at office / home. I thought it would be a bigger issue when I had the SE.

Same diameter doesn't mean same area. Different aspect ratios can cause differences in sizes anyway (this was used by the PC industry at the time when they switched from 4:3 to 19:9. Same diameter, smaller panels, ads looked better).

So yes, both phnes are 5.8", but S10e is 82.8cm^2, while XS is 84.4 cm^2 (XS: larger display). S10e has 83.3% screen-to-body ratio, XS 82.9% (XS: larger body, the display takes less space than the proportion of the body size differences).

TL;DR: despite the same diagonal, S10e is slightly smaller than XS.