Best sources to start accumulating? Just ebay?

Likely Facebook Marketplace or maybe Craigslist. eBay is pretty rough these days by all accounts.

eBay, yard sales, Goodwill, local thrift stores, Facebook Marketplace, etc.

For specific music Discogs.org is a very good marketplace. It's not free, it's a second hand marketplace, people don't give away things, but prices aren't worse than retail except for actually rare collector's stuff. But I've used it a lot to fill in the gaps in some artists' discography. Also older CDs are usually mastered with a bigger dynamic range, so they sound better on music hardware with a functioning volume knob. ;) So I much prefer buying original CD releases versus re-releases of an album on CDs anyway.

Also: Your relatives and friends. And their friends and friends' friends. Your neighbors. Ask around. And whenever someone moves, they usually want to get rid of things.

Buying used media could also a viable option. My kid likes to watch the same thing over and over, subscription content does not make sense for her. You just have to buy it slowly.

But didn't like 99% of people do a dead stop on buying physical media before 4k became a thing?

If there's something truly rare and hard to find, sure I'll watch it in some low-res format, but when it's online in 4k...