Usually there are other witnesses, other video sources. The more material exists, the harder it is to make the fake believable. You have to consider the sources. Who published the video, what are their motivations, who pays them.

Really, that's "internet user 101", I can't believe we're having such a discussion on HN :)