In your example, non competes seem to be optimal for the hedge fund investors, but I fail to see how it was optimal for everyone.
California has been pumping out quite a stream of “interesting jobs” and interesting businesses in a world of no non competes.
It was also optimal for me as the employee because the fund could (a) hire me and (b) structure my role in a way that I could quickly learn and make impact which (c) enabled me to earn what I want.
Without the non-compete, the role/hire would not have been possible and would have to be structured in some less optimal way.
Like I said, if I didn't think this role was even w the non compete I'd have done something else.
It's pretty standard now for employers to spring on you a previously-unmentioned non-compete clause at the time that you're onboarding. I've learned to ask for it before accepting an offer but I don't think most employees know to do that.
I'm a hardcore capitalist and I think non-competes have a very valid purpose in a limited set of circumstances. But the ubiquitous use right now is to make it harder for employees to leave.
There was no "money for the non-compete" involved, other than perhaps the premium baked into my compensation to begin with to make the deal attractive.
Think about sota deep learning models. Much better that everyone knows about hot chat gpt works.
However you don't get to dictate what the people who have actually invested the resources get to do with them, including what conditions they attach to who can be exposed to it (a la the non-compete). As an adult, I retain the option to not take the job with the non-compete.
In the 2 years I worked in the fund, I learned a tremendous amount thanks to being exposed to the "secret sauce" which would have been really damaging for me to take to a competitor.
After I left the fund, I took the 2 year non-compete (ie: don't work in another fund) to explore other industries, ended up in a FAANG and quite happy.
Point being, as an adult I weighed the risk of the non competes vs the opportunity of the role and made a decision. I am not sure how this kind of role would work in a world of no non-competes but I suspect it would be suboptimal for everyone.
The bottom line is, a non compete is not something that springs on you. It's something you evaluated in context and chose, as a professional. You don't have to take the job.