Re:Meta developer tools: Working at scale
by barbazoo on Wednesday June 28, 2023 @10:54AM
(#36507355)
My current role for the first time exposed me to amounts of data and variety of data (in type and how/where it's stored) that's difficult to learn organically. I wonder how a FAANG does that at "scale".
> most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts.
Is that the kind of question you're talking about here?
For many, the world has moved on a lot since, and those tools feel obsolete but so embedded in practices that it’s unthinkable to apply better approaches. One example of that is tracking data sources: legacy is slow, dysfunctional, and even fairly straightforward questions time out because you have to load so much meta-data. That’s because no one really uses it: most of that information is carried by the Data engineers who have built the system, and they are over-whelmed with questions that could be answer with a good system, but derive power over analysts. They benefit from it through comments they copy in their review, so fixing it isn’t anyone’s priority.
Others have evolved because the internal demand pushed things forward. PMs want Deltoid to work, so that system has moved to be state-of-the-art or interestingly unique in many ways: scaling, not implementing MCC, integration with the metric definitions.