The vacuum ISP of a N2O monopropellant is given on Wikipedia as up to 180s, but the cartridge weight drops that to 48s. One could be chilled with dry ice to a vapor pressure of 2bar, but it's still far from easy to handle when weight is a substantial issue and you want to dynamically control flow, I'd figure.
Making a tank that's light enough to fly but does not turn into a pipe bomb is _one_ of the hard parts. Also, ignition is and fueling is difficult. iirc Tripoli requires remote fill on hybrid rockets and since a lot of these experimental engines are flown in the desert at FAR (https://friendsofamateurrocketry.org/) temp. becomes an issue.
people are doing it though, i mean it's really impressive to see how far amateurs can push things. We may see the first individual reach the Karman Line this September at BALLS if Covid19 doesn't get in the way. A couple of college teams have done it but not just a random guy. Granted, the guy I'm betting on is very well known in the hobby and not "random" at all.