Re:Go Python, Go: Stream Processing for Python
by masklinn on Thursday October 12, 2017 @09:37AM
(#15457811)
Having to create a class called xPartitionFunction doesn't seem to favour easy pickability. Why isn't it just a function/callable?
Likewise builder, just ask for a callable, and the smallest "builder" is the state class itself.
Computations seem similar, the APIDoc states they must provide a compute() method but the example shows only a compute_multi. Is there a use case for having both on the same object? If there isn't, just ask for a reduction function and provide a decorator for one of the use case. Or don't and always ask for a list of results.
Or handle the "multi" case via a generator I guess (call reducer, if it returns a generator run it and add all items to the processing queue, otherwise add the one item).
Likewise builder, just ask for a callable, and the smallest "builder" is the state class itself.
Computations seem similar, the APIDoc states they must provide a compute() method but the example shows only a compute_multi. Is there a use case for having both on the same object? If there isn't, just ask for a reduction function and provide a decorator for one of the use case. Or don't and always ask for a list of results.
Or handle the "multi" case via a generator I guess (call reducer, if it returns a generator run it and add all items to the processing queue, otherwise add the one item).