I am not sure I agree with this: it seems to me that a quick ruling could be made on recusal and that it would arise quite rarely. But I'm not familiar with the system, and the lawyers I have heard from are. I suppose if judges are nearly always making good choices about when to recuse themselves the system should work fairly well.
This does depend on where you are, different jurisdictions will draw the line in very different places. E.g. England has particularly strict rules on bias and the appearance of bias in judges: in a famous recent case, Augusto Pinochet successfully got a ruling that he could be extradited to Spain overturned, because one of the judges on the five judge panel (Lord Hoffmann) was a director of a charitable trust linked to a body (Amnesty International) that had submitted an amicus brief in the case.[1]
(Not that it did Pinochet much good, since a differently constituted court later came to the same conclusion as the original one).
[1] http://www.parliament.the-stationery-office.co.uk/pa/ld19989...