Icons designed to be small have less detail and often rely more on the general shape, sometimes exaggerating certain features of the icon to make them stand out. Large icons or images can be filled with lots of rich detail. Scaling up a small icon often makes it look barren and generally out of proportion, and scaling down large, highly detailed icons/images makes the icon "noisy" and loses the essence of the icon.

This does not only apply to vector icons, by the way. Most highly polished icon sets will have an icon for different sizes. For example, in Windows, zoom the desktop icons in/out and compare.