It's more complicated than that: Emacs packages should be thought of the same as any other language's packages. They might comprise multiple files, and it's possible that their performance will be noticeably compromised if they are not byte-compiled. Installing Emacs packages should be done with the package manager.