A couple of years back, I burned out a friend's tweeters by hooking up an Arduino's 31.25kHz PWM to his stereo system. Worked fine at first, but it eventually melted his tweeters. I figured the stereo would low-pass-filter the input for me, and I paid the price.

The PC speaker didn't have PWM; it had a square-wave generator with adjustable frequency. Very few 1980s programs achieved anything even approaching decent sound quality using the PC speaker, although by the 90s, CPUs were fast enough that it was reasonably feasible. In the late 90s, I used to patch my Linux kernel with an unofficial driver that provided arbitrary PCM output via the PC speaker. It was glitchy if I left interrupts disabled during IDE disk accesses, which was the default at the time due to some buggy IDE devices that would sometimes corrupt data if you didn't. Fortunately, mine wasn't one of them.

I feel like PDM is just as simple as PWM and often gives better quality, and an R-2R DAC like a Covox is arguably just as simple as PWM and gives dramatically better quality. You need a few more pins, though, or a 74595.