Such a fine can be challenged in open court, no? Or is there an order by a secret court preventing Yahoo challenging the fine in a regular court?

How do you defend yourself from secret orders issued by secret courts that forbid defending yourself? What is this, mafia or the government ?

Yahoo challenged it in FISA court, not open court.

"Yahoo took its case to the foreign intelligence surveillance court, also known as the Fisa court, which oversees requests for surveillance orders in national security investigations. The secretive Fisa court provides the legal authorities that underpin the US government’s controversial surveillance programmes. Yahoo lost its case, and an appeal."