This effectively neuters the ability for federal agencies to create regulations within the specific areas that Congress has tasked them with regulating. This is a significant blow to both legislative and executive branches, and further entrenches the power in the judiciary.

In both reality and your supposed system, someone could always go to appointed officials and get their patent enforced - just look at how the Eastern District of Texas operated for decades. In reality, we see far more frequent turnover and shifting of opinion in federal agencies than we do in the judiciary. Just look at how often Net Neutrality has flopped back and forth at the FCC. (Constant reversal of regulatory decisions is also an issue, but it goes to show that the idea that a patent system that exists outside of judiciary control wouldn't have plenty of opportunity to make your case to sympathetic ears is silly)

The agencies can draft any ruels they'd like and put them before congress. Perhaps we can have less rules...

This does not neuter federal agencies. They can still make rules, and they still have deference on fact questions. The only difference is that on questions of law, they no longer have deference.

philosophically I see what you're saying, but will congress be good at passing small laws to amend the ever changing regulatory landscape? I'm going to guess no. So practically this could be a nightmare

Except that instead of deciding a specific patent, now all you need is one judge to throw out "patents". Or "environmental law".