If these individuals are exceptions and the rest are competent, then there should be minimal impact beyond creating a side show. As a tiny minority, they wouldn't have any real sway in the bill construction within their party, especially if it has any chance of passing. The split between the parties is usually narrow, so the representatives with the most potential to influence passing legislation are the ones near the center margins as they may vote against party lines and provide a bigger base of support. The reps near the outside margins tend to have less influence. Even if they sway some stuff during the creation of the bill, it's stuff that likely has to get ripped out to find enough support to pass into a law.

But if things are passing by only 1 or 2 votes anyways, where anyone 1 persons vote is a major deciding factor, that's an indication of a bigger issue. Such a divide means that half the country feels they are getting screwed and there was little compromise to include their concerns.