Re:Streamz: Python pipelines to manage continuous streams of...
by cashsterling on Friday March 27, 2020 @01:24PM
(#22704824)
Is streamz/CuDF/CUDA a good option for soft real-time processing of high bandwidth sensor data?
For instance, I have electronics board grabbing 10,000 analog voltage samples a second on multiple channels.
I want to:
- ingest chunks of that data at 100ms intervals (so 1000 samples per interval)
- update some plots in a PyQt-GUI at the 100ms interval
- and once a second compute a FFT of 10-20 seconds of data for one or more channels of data.
I have basic code working using numpy, numpy.roll().
I am debating whether or not I should be using C#/.NET or C++ instead of python for this... there are strong speed advantages.
Numpy is fast enough to handle what I need to do right now... but if I want to up my data rate by 10X (10kHz to 100kHz) I think I am going to run into hard limitations with python and numpy and PyQt GUI updates.
For instance, I have electronics board grabbing 10,000 analog voltage samples a second on multiple channels. I want to: - ingest chunks of that data at 100ms intervals (so 1000 samples per interval) - update some plots in a PyQt-GUI at the 100ms interval - and once a second compute a FFT of 10-20 seconds of data for one or more channels of data.
I have basic code working using numpy, numpy.roll().
I am debating whether or not I should be using C#/.NET or C++ instead of python for this... there are strong speed advantages.
Numpy is fast enough to handle what I need to do right now... but if I want to up my data rate by 10X (10kHz to 100kHz) I think I am going to run into hard limitations with python and numpy and PyQt GUI updates.
Thoughts and suggestions are much appreciated :-)