See for instance this GNU page where permissive licenses like the MIT license are also described as "free software licenses":
"The term “open source” software is used by some people to mean more or less the same category as free software. It is not exactly the same class of software: they accept some licences that we consider too restrictive, and there are free software licences they have not accepted. However, the differences in extension of the category are small: nearly all free software is open source, and nearly all open source software is free."
In particular, he notes the example of DRM software. DRM software can be distributed under an open-source licence, but by design and in practice it serves to restrict the freedoms of its users, not to enhance them.
I think you misunderstand the distinction between open source and free software. Open source software also needs to be available for redistribution. The main practical difference is Free Software also requires that if you make any changes to the software, those changes must _also_ be released under the same licensing, whereas open source may or may not have such provisions.