Re:Ask HN: What does your ML pipeline look like?
by tixocloud on Sunday April 21, 2019 @07:06PM
(#19715126)
+2 for bringing the (ML/stats) code to the data instead of the other way around
Re:Ask HN: What does your ML pipeline look like?
by thiggy on Tuesday April 23, 2019 @12:46PM
(#19730367)
Could you speak to your experience with this particular list item?
Re:Ask HN: What does your ML pipeline look like?
by tixocloud on Wednesday April 24, 2019 @04:05AM
(#19736448)
We deal with fairly large volumes of data on a frequent basis so it would not make sense for each data scientist to create a copy within their own environment. Everyone works off a centralized data source and we provide them with Jupyter/Spark in an internal cloud environment.
As a scientist designing risk management systems, I also like to:
. avoid moving the data;
. bring the (ML/stats) code to the data;
. make in-memory computations (when possible) to reduce latency (network+disk);
. work on live data instead of copies that drift out-of-date; and
. write software to keep models up to date because they drift with time too and that's a major, operationally un-noticed, and extremely costly problem.
I'm not yet into Tensor/ML-Flow, but I use R, JS, and Postgres, thereby relying on open-source eco-systems (and packages) that are:
. as standard as possible;
. well-maintained;
. with a long expected support; and
. as few dependencies as possible.