Re:Data Engineers Spend 2 Days per Week Firefighting Bad Data
by morelisp on Tuesday August 16, 2022 @01:45PM
(#32486228)
What a weird article - poor data quality impacts 26% of their companies’ revenue - how in the world do you even begin to calculate something like this? It reminds me of "X% of revenue lost to software piracy!" bs. Gotta sell those "helpful" data products I guess...
There's always going to be a lowest quartile (or whatever) quality of data. Remember the recent SRE thread? You should be modeling "data quality budgets" like "error budgets". It's also naturally corrective, if you spend less than Xd/w on firefighting you're going to start scaling up/out features or scaling down costs until you are spending Xd/w dealing with data fires again.
In the end, data engineers exist because "bad data" exists. That's the job.
Re:Data Engineers Spend 2 Days per Week Firefighting Bad Data
by rgavuliak on Friday August 19, 2022 @05:30AM
(#32519024)
I could imagine a simple use case where the revenue reported is off by 26% and bam, 26 % of revenue impacted. You could then go into revenue attribution for another revenue impact.
There's always going to be a lowest quartile (or whatever) quality of data. Remember the recent SRE thread? You should be modeling "data quality budgets" like "error budgets". It's also naturally corrective, if you spend less than Xd/w on firefighting you're going to start scaling up/out features or scaling down costs until you are spending Xd/w dealing with data fires again.
In the end, data engineers exist because "bad data" exists. That's the job.