It's not a scam for you but rather a scam for society at large. Making money purely by owning some revenue generating asset, while not providing any service yourself is doesn't work if every single person does it. Many people who work at companies on the S&P500 aren't paid enough to having savings to invest. The money the company saves on employee salaries are shown as profits which result in the "passive income" that you're making.

I don't 100% agree with the logic because it doesn't account for improvements in productivity but I think it roughly translates especially in some late stage capitalism cases where company profits grow despite lower productivity (eg. layoffs).