I think with this amount of money it's just a risk of funding people. Putting in strings requiring repayment if the person changes areas, when you're giving someone <$100k, is pretty heavyweight for either the NSF or Y Combinator, so they just try to pre-screen for people they think are going to use the money as intended and stick with it, and accept that some percentage of people will instead use the money to flail around for a year or two and leave. If the percentage gets too high, then I guess reconsider the screening process.
I think you're right re: clawbacks, though. I can't imagine the mental pressure were the scenario to be "you MUST stay in academia or you owe the government $XXXk". I actually know a PhD student on a (non-US) government scholarship who has a clause like that. He calls it his "slave contract".
Getting back to the original topic, I think that for any venture with a high rate of failure (startups or grad school or...), there has to be some room to allow the funded person/people to say "this isn't working" in good faith and pivot or leave. That's totally different from someone taking funding without the intention to use it properly, or just deciding they're bored or whatnot.