Re:The Dawn of the Data Intelligence Engineer
by jghn on Friday November 13, 2026 @10:40AM
(#50485115)
Keep in mind that the exact same arguments were being made 15, 20 years ago about why specialization was important. And yet, we look back at that era being the halcyon days of when specialization wasn't necessary.
Every era views itself as an intricate snowflake, but they rarely are.
Re:The Dawn of the Data Intelligence Engineer
by bunderbunder on Friday November 13, 2026 @10:55AM
(#50485375)
I think that might be the goomba fallacy? There is no universal “we”. Some people think specialization is important. Others think generalists are important. Others talk about T-shaped skills. These are potentially all different groups of people that ideally realize that there are tradeoffs to each option and the best compromise is context dependent.
And I do think it’s more about how you build things than it is about the calendar. The calendar is just a proxy for what’s fashionable. For example, the startup I currently work at is in the process of incrementally simplifying our stack. Because we agreed that it was getting difficult to understand and expensive to operate. We don’t view this as turning the clock back to 2006; the discussions have framed it as a judicious application of the KISS principle. And yes, it has helped us break down knowledge silos and enable team members - especially data scientists - to be more self-sufficient.
15, 20 years ago, me and my T-shaped skills could get by just fine with a database server, a NAS, and some frank conversations about priorities.
Nowadays it’s not enough to to know how to work with databases and files. There’s also all these platforms: databricks, snowflake, new relic, sentry, datadog, GitHub actions, prefect, dbt, kubernetes, cassandra, elastic, and on and on and on. Each brings a new set of idiosyncrasies, its own terms for things, its own enterprise integration headaches. It’s just a hell of a lot of knowledge to as one person to cram into their head on top of whatever skills and domain expertise form the actual core of their job.
That makes sense for an enterprise. Enterprises have a lot of bodies to throw at the problem, so they can have an expert for each tool who’s available to support everyone else. And they like to have support contracts and SLAs, which you can get from à B2B vendor but not from the KISS principle. And they don’t mind great heaping piles of unnecessary and mutually conflicting abstractions because that sort of thing is just SOP for a large bureaucracy.
But if you want - need - a system that one person with T-shaped skills can reasonably fit into their head, this is not the way.