You're treating that theoretical average or median, even if no one speaks that way, as the standard accent against which all others are compared.

That's one way in which people sometimes choose a "right" or "zero" accent. Another is by social standing ("the queen's English" was literally that).

Either way, that's declaring one accent the right way of speaking, and describing people who speak differently as having a "heavy" accent instead of simply a different way of speaking.

Perhaps a specific example would make this clear. There are many dialects and accents of German in Germany. Which one would you declare the standard, and who would you describe as having a "heavy" accent?

My point is that making such a judgement is wrong to begin with, and one should simply describe a particular accent by its features or regional distribution, not as "heavy".

(And to respond, briefly, to the comment below: when learning a foreign language you choose to learn a particular dialect and accent. That doesn't make other accents "heavy", they just aren't the accent you've learned.)

It declares it standard, not "right". If you learn a foreign language having a standard to aim for is good and large deviations are difficult. Stop trying to turn this into right or wrong.

My whole point is to avoid turning this into right or wrong by realizing that an accent or a bias merely describes a particular way of speaking or thinking, not a degree of deviation from some standard.

In linguistics, this is the prescriptive vs descriptive debate. It's not controversial that a prescriptive grammar declares how a language should or ought to be used. Declaring one accent the standard is a similar prescriptive position.