That's like saying that the freedom to repair your own car does not apply to car owners because almost nobody is a car mechanic.

The free software freedoms are what allow any end-user to freely choose anyone they want for audits, repairs and extensions of their software, instead of being dependent on a monopolistic supplier. Whether the individual user has any clue of software development is completely irrelevant.