That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things.

Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.

I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.

I am not sure how this answer my question. I understand that a feature store contain the features that have signal.

My question is about the operational aspect of online vs offline stores.

I.e. what happen if the features needed for a prediction are not in the ONLINE store?

You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit:

https://databricks.com/session_eu20/real-time-feature-engine...

Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.

If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.

No, the feature would have to be in the online feature store (NDB, not Redis in our case). We have metadata about the schema of the model (the same schema as its training dataset) so that the applications can just supply the IDs and get back the feature vector in the correct order and get it also in the format needed (a npy array or a tfrecord example). This really helps maintenance, as adding a new feature means you don't have to update the application to account for the new feature and ensure the ordering of features matches the ordering in the training dataset.