The time in the RNN might be the only thing that's hard. We can commonly do A/D, D/A pretty quick. I can make a db request over a network, have it parse the SQL, execute it reading a bunch of SSD pages and return the sorted results in about 1ms.

The answer would probably be to reduce the 'learned' output to be a convolution kernel that gets run rather than the RNN itself on the input. Then the kernel only has to change gradually to produce a different sound not continuous processing to produce a particular sound.