That’s mostly accurate, but the nuances are significant and lead to different conclusions. For example the hypolimnion may be much warmer than 4C in lakes in warmer areas. More importantly tropical ocean water is above 4C down to roughly 2km and not only is that depth expected to increase, but also the depth of warm water as you go north. https://en.wikipedia.org/wiki/Thermocline

The important thing to remember is even a 1 part in 1,000 decrease in density * 2km of depth = 2m of expansion. Ballpark estimates aren’t enough you really need fairly detailed simulations to get any significant accuracy. Actually doing such simulations shows meaningful sea level rise from thermal expansion at ~0.07 inches per year or roughly half the current rate of increase. This might not sound like much, but consider that volume of sand you need to replace to maintain beaches etc etc.