If the judge gets suspicious, you'll have to explain the potential issue to him off the record. After that you can only continue to say it if the judge thinks that you have a potential reason for worry. But they err on the side of respecting it.

That said, a defendant who pleads the fifth at all questions will not necessarily help their case...