As for incompetent knights, I'm genuinely confused: how did they survive in the role for any length of time? (Especially when the only talent pool they would have to recruit underlings from are the very same people they're oppressing. I always figured competence at protecting the outlying villages—where everyone's families live—during invasion, was kinda the reason anyone bothered to work for them at all, rather than towns responding to recruitment attempts exactly the same as invasion attempts, with stoning, boiling oil from the rooftops, etc.)
The villagers, on the other hand, are used to invaders coming through. Their condition is likely to remain poor either way: The knight is more likely to come through after something happened, rather than preventing it from happening. Most of the daily decision making is up to the community.
The knight's recruitment efforts can always come with the promise of gaining wealth and status, which will always encourage some young folks, even if the knight is a bad boss who plans no real rewards. Records that uncover his lies are hard to acquire and most of what you learn will come from rumor instead.
Not all knights saw that much action. I agree though that they had strong incentives, besides survival in battle, to be at least defensibly competent.
As to those incentives -- they had a multi-generation arrangement with the sovereign: to supply a certain number of mounted men when he called for it, maybe determined by the size of landlord they were, and in return the king let them keep dominating economic activity in their usually-rural area. The knight held the purse strings and could extract as much as he felt prudent from his tenants (farmers etc).
But the knight would employ Men-at-Arms. These were the guys with what we think of today as 'Knight Skills,' but no land/title/purse. They'd be taken care of, and they'd definitely try to keep their boss alive. Or sometimes an older knight might send the required men without going himself. (This varies a lot by time period of course).
So on the one hand, they didn't have to do superhero stuff if they didn't want to.
On the other hand -- their whole reason to exist was intensely martial; the structure of society reflected the fact that folks riding horse-tanks dominated all of civilization. There were good reasons for someone with a coveted horse-tank position to, yes, be able to be ride, be at least minimally competent, and show up to risk their lives when the king calls for it, or else risk losing their privileged position.
> the reason anyone bothered to work for them at all, rather than towns responding to recruitment attempts exactly the same as invasion attempts, with stoning, boiling oil from the rooftops, etc
Well, the knight owned the siege-proof fortresses in the area; that's one of those things they did well in feudal times, picking geographical strong points for fortresses.
The knight also owned the town people lived in too, the farms, houses, etc. His family got them way back when kings started deciding that they didn't want the town to send 5 random foot soldiers when he called -- he'd rather have 1 professional, well-equipped horseman with mail, shield, lance etc, and if you could make that happen he'd just give you your neighbor's stuff to keep making it happen in perpetuity.
Most knights didn't fight often - if at all. Even when called upon they could just supply another person instead of themselves.
More interesting questions are:
Was the average knight fat?
How common were the?
How effective were fat/incompetent knights on the battlefield?