I think the problem is rather that with the need for taxation you also have to introduce mechanisms to check if taxation actually happens properly and with that comes a whole rat tail of bureaucracy and regulation complicating the process for everybody.

In theory, exemptions can be made, like only taxing people who grow for commercial use while not taxing people who grow for their private use. But such a system is prone to abuse and thus needs lots of control and enforcement.