This effectively neuters the ability for federal agencies to create regulations within the specific areas that Congress has tasked them with regulating. This is a significant blow to both legislative and executive branches, and further entrenches the power in the judiciary.

In both reality and your supposed system, someone could always go to appointed officials and get their patent enforced - just look at how the Eastern District of Texas operated for decades. In reality, we see far more frequent turnover and shifting of opinion in federal agencies than we do in the judiciary. Just look at how often Net Neutrality has flopped back and forth at the FCC. (Constant reversal of regulatory decisions is also an issue, but it goes to show that the idea that a patent system that exists outside of judiciary control wouldn't have plenty of opportunity to make your case to sympathetic ears is silly)