The average depth of the ocean is about 4km, so a thermal expansion of 1x10^(-4) per degree C (right order of magnitude), would result in about a 4m rise per degree increase in average ocean temperature. Which is obviously significant.

One of the issues with this theory is that the average temperature of the ocean is below 4 deg C, and in this regime the coefficient of expansion is negative. So we might actually see a small sea level drop due to ocean warming.

While there is probably a slight amount of expansion in liquid water from heat, water expands most when frozen solid. this is the entire reason ice floats

Half of all recent sea level rises have been from thermal expansion of water: https://www.nationalgeographic.com/environment/global-warmin...