https://pysdr.org/ is a good introduction to DSP/SDR using Python.

Previous discussion on HN: https://news.ycombinator.com/item?id=24750588.

Ham Radio Workbench podcast on GNU Radio: https://www.hamradioworkbench.com/podcast/GNU-Radio.

In terms of hardware, there is a large spectrum, from inexpensive RTL-SDR (receive only, ~$25), PlutoSDR (transmit and receive, $229, which PySDR covers) all the way to $x000+ for USRP etc.

There are also cheap SDR transmitters:

https://osmocom.org/projects/osmo-fl2k/wiki (repurposing a USB-VGA dongle as analog source)

https://github.com/F5OEO/rpitx (repurposing bitbang/PWM GPIO; we had weird problems with data corruption when we were trying to use it as a radio modem, but maybe you will have more luck)

https://bellard.org/dvbt/ (repurposing standard VGA card, but it's probably not worth it since fl2k is way better)

(beware that low quality of the transmitter usually means it will cause interference with other stuff. However, all of these have such a low power that if you will not use an amplifier, it will be OK, the interference will be probably undetectable outside of the room where the transmitter is)

I would suggest these two as a crashcourse to the math that is used in SDR:

https://www.dsprelated.com/showarticle/192.php https://www.katjaas.nl/home/home.html

We are maintaining a long list with theory/math: https://brmlab.cz/event/dsp#zdroje , and engineering stuff: https://brmlab.cz/project/sdr/start#links . However, I agree it is rather difficult to get into the topic. I'm playing with SDRs for almost 10 years (increasingly "fulltime" lately), and we still need to employ a professional mathematician to help me with some advanced problems.

Check out the Field Expedient SDR series. They offer a decent introduction for beginners.