The issue with panels allocating taxpayer money, is that their interest is still in government. That is, they will fund research in which government has an interest or is neutral, and consequently, any research which is potentially harmful to government is scrapped (even if it would be beneficial to the taxpayer.)
One might not think there's a distinction between what benefits the government and what benefits the taxpayer if one believes in the fairy tale that governments exist to benefit the people.
Really? Am I really reading this?
Were the panels doing their duty when they awarded Felisa Wolfe-Simon her grant? Was the panel doing their duty when Leo Paquette, sitting on one, kaiboshed a young researcher's grant and stole the idea for himself? How about the panel that awarded a grant to Geoffrey Chang, in spite of having six retractions due to careless (and quite frankly stupid) crystallography? Is he really "someone who deserves it the most"? This is just the tip of the iceberg. I could list about 20 more off the top of my head.
I would go so far as to say as a rule they do not allocate scarce funds to whomever deserves them the most.