News for hackers. Stuff that matters.

A comment on Chronon, Airbnb's ML Feature Platform, Is Now Open Source. Parent comment

Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by nikhilsimha on Tuesday April 09, 2024 @03:19PM (#39983131)
We haven’t tried materialize - IIUC materialized is pure kappa. Since we need to correct upstream data errors and forget selective data(GDPR) automatically - we need a lambda system.

Tecton, we evaluated, but decided that the time-travel strategy wasn’t scalable for our needs at the time.

A philosophical difference with tecton is that, we believe the compute primitives (aggregation and enrichment) need to be composable. We don’t have a FeatureSet or a TrainingSet for that reason - we instead have GroupBy and Join.

This enables chaining or composition to handle normalization (think 3NF) / star-schema in the warehouse.

Side benefit is that, non ml use-cases are able to leverage functionality within Chronon.

[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by jamesblonde on Tuesday April 09, 2024 @03:57PM (#39983565)
FeatureSets are mutable data and TrainingSets are consistent snapshots of feature data (from FeatureSets). I fail to see what that has to do with composability. Join is still available for FeatureSets to enable composable feature views - join is resuse of feature data. GroupBy is just an aggregation in a feature pipeline, not sure your point here. You can still do star schema (and even snowflake schema if you have the right abstractions).
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by jamesblonde on Tuesday April 09, 2024 @03:58PM (#39983582)
Normalization is a model-dependent transformation and happens after the feature store - needs to be consistent between training and inference pipelines.
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by nikhilsimha on Tuesday April 09, 2024 @04:31PM (#39983925)
Normalization is overloaded. I was referring to schema normalization (3NF etc) not feature normalization - like standard scaling etc.
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by jamesblonde on Tuesday April 09, 2024 @04:49PM (#39984092)
Ok, but star schema is denormalized. Snowflake is normalized.
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by nikhilsimha on Tuesday April 09, 2024 @06:10PM (#39984841)
To be pedantic, even in star schema - the dim tables are denormalized, fact tables are not.

I agree that my statement would be much better if used snowflake schema instead.

[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by jvican on Tuesday April 09, 2024 @05:19PM (#39984374)
Thank you for sharing!
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by throwaway2037 on Tuesday April 09, 2024 @11:04PM (#39986566)
What is the meaning of pure kappa?
[ Parent | on HN ]
Re:Chronon, Airbnb's ML Feature Platform, Is Now Open Source by pedrosorio on Tuesday April 09, 2024 @11:46PM (#39986756)
https://learn.microsoft.com/en-us/azure/architecture/databas...
[ Parent | on HN ]