News for hackers. Stuff that matters.
Re:Machine Learning in Production (CMU Course) by belter on Tuesday January 28, 2025 @07:14AM (#42851462)
This seems to have very little on Data Quality and it is on Chapter 16...How much practical experience in Industry do the authors have? Because 90% of your time will be spent on Data Quality and Data Cleansing...
Re:Machine Learning in Production (CMU Course) by szvsw on Tuesday January 28, 2025 @10:10AM (#42853178)
Arguably that’s a separate (obviously critical) concern. I think it’s worth it to abstract that away as just a step that exists in the pipeline with its own set of concerns/challenges/methods etc that really requires its own deeper study to do well.

For instance, my ML work is almost entirely in the context of engineering simulation regression/surrogate development, where data quality/cleaning is almost no issue at all - all of the work is on the dataset generation side and on the model selection/training/deployment side.

Every job is different!

Re:Machine Learning in Production (CMU Course) by bigmutant on Tuesday January 28, 2025 @02:57PM (#42857161)
Agree, Data Quality in-the-wild is a huge concern. I've led efforts to establish Lineage/Quality in large orgs and doing this after-the-fact is a massive undertaking. Having this "up-front" before all the data pipelines (origination, transformation, pre-processing) calcify saves a lot of headache down the road.