The problem is beliefs aren't homogenous. If 99% the world that tesla was worth $0 and 1% thought it was worth $100, then without short sellers the price would trade much closer to $100 than 0. But if there are short sellers than the beliefs of the market start to average out more.

Also there is empirical proof short selling matters because during the banking crises they banned short selling which causes certain bank stocks to jump up, and then when they allowed it again the prices fell like a rock.