Empowering data scientists with a feature store

Re:Empowering data scientists with a feature store by tronbabylove on Sunday October 10, 2021 @11:50AM (#28819324)

Interesting, thanks for sharing.

How do you handle historical backfill for new features? As in, some feature that can be updated in streaming fashion but whose initial value depends on data from the last X years, e.g., total # of courses completed since sign-up.

Also, who is responsible for keeping the Flink jobs running: the data scientists, or do you have a separate streaming platform team?

[ Parent | on HN ]

Re:Empowering data scientists with a feature store by yiksanchan on Sunday October 10, 2021 @10:28PM (#28824001)

Thanks for your kind words!

> How do you handle historical backfill for new features?

Currently, our feature store doesn't come with inbuilt feature backfilling. In order to do that, some manual work needs to be done. We're working on a brand new version of feature store that hopefully addresses this need.

> Who is responsible for keeping the Flink jobs running: the data scientists, or do you have a separate streaming platform team?

We have a separate data infra team who is responsible for managing the YARN cluster for us.

[ Parent | on HN ]