> people simply MUST be doing self-guided experimentation

And self guided exploration is a skill in itself which you have to learn. You can experiment for years and get nothing of it because you don't even measure anything. You can find a local maximum and, not knowing the concept, never try something radically different.