Re:Machine Learning Engineer Guide: Feature Store vs. Data W...
by streetcat1 on Thursday October 08, 2020 @11:17AM
(#24719879)
I am not sure how this answer my question. I understand that a feature store contain the features that have signal.
My question is about the operational aspect of online vs offline stores.
I.e. what happen if the features needed for a prediction are not in the ONLINE store?
Re:Machine Learning Engineer Guide: Feature Store vs. Data W...
by jamesblonde on Thursday October 08, 2020 @12:38PM
(#24720837)
You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit:
https://databricks.com/session_eu20/real-time-feature-engine...
Re:Machine Learning Engineer Guide: Feature Store vs. Data W...
by skingkong on Thursday October 08, 2020 @11:50AM
(#24720277)
Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.
Re:Machine Learning Engineer Guide: Feature Store vs. Data W...
by jamesblonde on Thursday October 08, 2020 @12:39PM
(#24720847)
If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.
Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.
I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.