That isn't a good way to measure because the inside of the house is warming up the walls. Hardwood is a poor insulator, but it is an insulator. If you touch wood it will feel warm, but it still loses a lot of heat compared to a properly insulated wall.