Re:Easily Create Complex Workflows with AWS Step Functions
by forrestbrazeal on Saturday February 25, 2017 @04:26PM
(#13733926)
Right now, we have to coordinate our own batch processing, since dynamic parallel states aren't supported, as you note. I've actually asked the Step Functions team about this feature, and I know I'm not the only person to do so, so hopefully it's on their roadmap :)
Re:Easily Create Complex Workflows with AWS Step Functions
by vrudyi on Saturday February 25, 2017 @03:58PM
(#13733789)
I think there may be better options than Step Functions(or that can be used from Step Functions) for this particular purpose. Depending on the processing required, passing data further to something like recently announced AWS Batch can solve such problem.
Such as: a parent state's output will determine the size of the input to pass on to multiple copies of identical parallel state children (the number of which is determined by the parent's output). These get the same input or a specific slice of the same input from their parent.
I guess these could be created dynamically within the parent, then deleted afterwards. Not as clean as declaring this in the states markup though.