This argument came up during the Affordable Care Act debate. The question was, if the feds can make you buy health insurance, could they also make you buy broccoli? There's this famous exchange from Justice Kagan's confirmation hearing:

  Senator Tom Coburn, Republican of Oklahoma, who is also a medical doctor, asked, “If I wanted to sponsor a bill and it said, ‘Americans, you have to eat three vegetables and three fruits every day,’ and I got it through Congress and it’s now the law of the land, got to do it, does that violate the commerce clause?”

  “Sounds like a dumb law,” Ms. Kagan replied.

She declined to presume that it would be unconstitutional. Justice Ginsburg had a similar interpretation.

https://scholarship.law.bu.edu/cgi/viewcontent.cgi?article=1...

They believe the constitutionality of enforcing such transactions depends on the details of its application, and is not facially unconstitutional. So if the legislature could be convinced that pot is health supporting, SCOTUS may well find it to be within their power to mandate.

The idea that the constitution doesn't allow the government to force me to say something, but might allow them to force me to put something into my body, is completely wild to me.

The argument is that by growing your own marijuana you affect Congress's ability to regulate the interstate commerce of marijuana, which is consistent with Filburn.

But where does that end? Can Congress prohibit you from earning an income because you might use that income to buy marijuana?