A lot of sellers on etsy couldn't afford the $1k to start.

That's why I had the (replace by magic numbers) in parenthesis. There may be some number where you can still attract legitimate sellers but is high enough so people taking a risk to just peddle junk will turn a loss on average. The only way you get rid of the junk is to make their expected profit less than zero.

that would go over really poorly.

nobody would spend $1000 on what is essentially a gamble (since etsy is determining whether you're 'selling mass produced junk').

sellers already feel like etsy is way too buyer-friendly (at the expense of sellers).

Etsy has to invest effort in making sure that legitimate sellers don't have this gamble. They have to be sure they're only penalizing the spammers. It's not unlike insurance fraud, do you feel it's a gamble that you're getting insurance because the insurance company won't pay out or will pursue you for fraud? Probably not. Yet they still try to identify and prosecute the fraud. If the site is dominated by spammers selling stuff from the dollar store everyone loses. You have to make it so that it's not worthwhile for the spammers. A deposit is one way. Pursuing legal action for contract violation could be another. Another industry with similar issues is banking.

It's not easy but you have to invest in maintaining the credibility of your platform. Machine learning helps but these problems have existed pre machine learning and can have more traditional economic solutions.

EDIT: The deposit could also depend on your prices and sale volume. This way really small players can pay in with a lower deposit. There are probably many variations but again the key is to make it a losing proposition for the spammers/dollar store guys assuming you want them off your platform. If you're only doing machine learning/filtering the cost to attack the system and try to get in is too low.