Similar to "chink in one's armor. A "chink" means "a narrow opening or crack", and a chink in one's armor means a weak point that makes you vulnerable to attack.

But "chink" is also an offensive term for a Chinese person, so "chink in one's armor" can be misunderstood, especially if what you are talking about actually involves a Chinese person. ESPN got in trouble a couple of times with it, even though they had used it thousands of times before without incident, because of a couple articles where the weakness they were describing on some team was a player who was Chinese.

Niggardly is another such word. It has no relation to the racial slur, originating in the Old Norse "nigla", but it would be unwise to use it today.