The downside in the extreme is that the original investor pulls the termsheet and you end up empty handed. I know of cases of this happening, but it's rare.

More commonly, you develop a bad reputation in the VC community, which is very close-knit. Next time you come to raise money, this can hurt.