We've been storing our features in RDBMS/normalized with each feature table having a run_id column (run_id is unique to pipeline version + timestamp of execution if batch processed; if streaming there is run_id for pipeline version but the date comes from interaction date, which is part of the raw data already]) and I'm curious what we're potentially missing..
In this sense you can query features for given users generated by particular version and/or date but it does involve potentially lots of joins (to get a collection of features).
I can see RDBMS being reasonable for small-to-medium size of data, but beyond a certain threshold, I think it starts to breakdown (at the few 10s/100s of TB level, maybe?).
...what data warehouse/database are you using that can't support those queries on-demand at low-latency and granularity?
It seems to me that these 'feature stores' are just tables of pre-computed common aggregates, optionally with some kind of versioning id/timestamp. In which case, why do I need some special service to do this, can't I just make a second database in my warehouse and write something that just computes all desired aggregations periodically and dumps them into the second db? Even easier, just use stored process and distribute them to data scientists, and use timestamp parameters to control versioning.
Semi-/un-related anecdote, but looking back at what my team was like 6 years ago, we used to be a classic enterprise data warehousing team, and our worldview at the time was basically that, "everything is just a bunch of databases or file systems, and our job is to copy (literally scp) files from source A to destination B." All problems (literally, all) would be solved as one of, "how can I export a file from this source, and what kind of table do we need to create in the DWH?" By that logic, we would make decisions like, "why do you need Splunk/Promethus/Datadog/etc. for operational monitoring, when you could just write those metrics to tables in the DWH?".
In retrospect, what is surprising about such a worldview, is not that it was so limiting or that it hurt us in the long run (it did, but less than one might expect), but how far you could actually go and how much you could actually achieve, with such simple primitives.
Do you have any particular pitfalls you could share that others might benefit from, mis-steps you made in this journey we might avoid? What really sucked about this process?
One milestone for success I think is, when your feature repository can be used to regularly gleam new insights that are surprising (even to your experts). So for example, imagine you are a typical retailer selling widgets, and you have 10000 customers who bought widget X, and you want to identify out of your remaining 10 million customers, which of those customers should you target as likely to want to also buy X? Now, of course, this is a classic problem and doesn't require a fancy feature repository; just the basic order information of who-bought-what is enough to create the classic collaborative filtering solution from.
But with a rich/useful feature repository, you can throw this problem in there, looking for the most similar other customers compared to the 10000 you know, and to rank which features are most powerful (in terms of, why it makes them similar), and you should see some interesting insights that you never would've guessed on your own.
Just to answer the "what really sucked" question a little bit, even if it's implementation-specific... man it really sucked to start out with 2-out-of-5 level of knowledge about Apache Spark, and to stumble through learning both the obvious lessons and the non-obvious/esoteric bugs/quirks, until after 1-2 years the team finally felt comfortable with like an 4-4.5/5 level of knowledge about Spark. This was also specific to the state of Spark back in ~2017 or so, and it's come a long way since then.
- Years and years ago, we already had a data warehouse (DWH)
- In the data warehouse, you would store data like, each and every order that all customers have made (i.e. up to and including full-fledged facts and dimensions about each)
- Now, let's say axiomatically/hypothetically, a very useful and highly predictive feature for ML is "# of orders made in past 7 days" for each customer
- Can this be computed from the data already in the DWH? Yes, absolutely, but it's a new computation and not an existing column/attribute in the dimensional model.
- What if you need to recompute this feature daily, for millions of customers and orders? Well, we could always just add it to the dimensional model, compute it once, and let people just use it/share it... but why? Most internal users of the DWH probably don't care about something like "# of orders past 7 days" as something to be added to a customer dimension or per-customer grain (too specific or whatever), and moreover, the DS/ML folks want the same feature but for "every 1/3/7/30/90/180/365/730/etc." days breakdown, as well as a bunch of variations about orders and things other than orders (e.g. "average time between new orders, over past 7/30/90 days" or "average $ spent over past 7/30/90 days", as features that serves as a proxy for frequency of activity and level of engagement)
- Hence, it makes sense to keep the "golden copy" of data in a canonical form in the classic/standard DWH as a baseline, and to separately/independently compute features out of that data and to store them in a different system (which can also be optimized for the different query/access patterns that DS/ML have, vs traditional BI). Over time, it also made sense in certain cases to go upstream of the DWH to source data from and process it more directly (for performance/efficiency reasons), though generally deriving features out of the transformed dimensional models was still very useful.
- It took our teams ~1-2 years to really go through this evolution and reach a mature-ish state, but for the past 2-3 years, we've benefited tremendously from having an independent feature repository/store, that is separate from the classic DWH. Benefits came in all the obvious and some non-obvious ways, i.e. in faster iteration/cycle time, in better quality/repeatability, and in being able to automatically discover interesting relationships that no human could have anticipated - simply by having a very broad/large repository of features and running automatic feature selection over it.