For a given temperature rise, metals feel hotter than many other materials because they have a high thermal conductivity and enough heat capacity to deliver a lot of heat quickly.

As another though experiment, consider that you dread walking barefoot across cold tile floors but can bear to walk across carpeted floors in the same house. These two materials are at the same temperature.

Also consider aluminum foil you just pulled out of the oven. It's thinness runs contrary to the large thermal capacity of a solid chunk of metal- you can touch it immediately because you are such a large heat sink compared to it that it can't burn you even while it has only just started (rapidly) cooling from 350°F.