One thing you might find helpful is to prototype things with GNU Radio and a GRC flowgraph. I'm not sure that would be useful for running on the ESP32, but you could at least tinker around with signal processing tactics that you could implement on it.