They wouldn't have to buy it back immediately. Short sellers would borrow the stock from a long and pay a predetermined interest rate for however long they're borrowing it. They technically don't have to rebuy the stock unless they get margin called (if the collateral posted is no longer deemed sufficient for the counterparty to allow the borrowing position) or need to close out positions because of some risk tolerance threshold being met. Usually when this happens to a bunch of shorts at the same time they must all rebuy, driving prices higher, which is called a short squeeze, but it is usually determined by someone cutting losses on a short position or positive news coming out driving prices up.

That's equivalent to saying that buying a stock doesn't move the price up because you'll have to sell it eventually. A lot of people in long positions means a lot of buys that haven't been re-sold. A lot of people in short positions means a lot of sells that haven't been re-bought.

Actually, short selling has another nasty sting in the tail.

If there are few willing sellers out there when everybody goes to cover their short, you get a short squeeze - a feedback loop where the price increases, so people need to buy to cover shorts to avoid losses, which pushes the price up, which means more people need to buy to avoid further losses as the price rises and so on.

The original owner who lent the stock can also ask for it back on relatively short notice (say, because they are long and want to sell). Usually this isn't a problem because the short finds someone else to borrow from and uses that to cover - but if enough people do that then it can trigger a short squeeze even without significant price movement at the beginning, as happened rather dramatically in Porsche shares in 2008.

Shorting a stock is like renting a car for a year, and selling it - because you think you can buy another car in a year, and still make money after paying the rental company. This is possible in this example because all cars are identical.

Hopefully, the sale to start the position and the purchase to close it do not move the market themselves. People do long positions without moving the market (affecting the price). Sorry if I explained anything to anyone who already knew it.