You can do this by enabling developer mode in chrome://extensions, which lets you install from a directory.
You lose automatic updates though.
yeah the permissions and ecosystem are scary. I have a very small extension (1000-ish users), and even I get "monetization opportunity for your extension" emails maybe... every other week?
Sometimes I feel like the only reason it's not a platform-breaking problem is that most extension devs make enough money from their day job to not care about a quick buck.
It would be interesting if Chrome let you point an extension to a github repo (and tag or commit hash) and pull source from there.