He took prescription speed daily. And supposedly was sneaky about taking other drugs. So yes, he mostly didn't openly participate in the drug scene. But it's less clear what he did privately.

http://www.warholstars.org/warhol/warhol1/warhol1c/warhol1cl...