The average depth of the ocean is about 4km, so a thermal expansion of 1x10^(-4) per degree C (right order of magnitude), would result in about a 4m rise per degree increase in average ocean temperature. Which is obviously significant.

One of the issues with this theory is that the average temperature of the ocean is below 4 deg C, and in this regime the coefficient of expansion is negative. So we might actually see a small sea level drop due to ocean warming.