It's weird to me that people build libraries on top of the ML stack to track provenance, when it's really the ML library's job to do that for its inputs. However it is a right pain building it into the ML library as it affects all the interfaces. We build data, model & evaluation provenance objects into our ML library, Tribuo (https://tribuo.org), as a first class part of the library. You can take a provenance and emit a configuration to rerun an experiment just by querying the model object. It is built in Java though, which makes it a little easier to enforce the immutability and type safety you need in a provenance system.

edit: I should add that I'm definitely in favour of having provenance in ML systems, and libraries layered on top are the way that people currently do that. It's just odd that people aren't working on adding that support directly into scikit-learn/TF/pytorch etc.